Our goal to unify fastlane tools

I’ve been working as an iOS developer for over 5 years for various companies around the world. During that time it became clear there was lots of room for improvements around mobile developer tools, and in particular, the deployment of apps. Because of that, I started implementing little tools to help me automate tedious tasks like uploading metadata for release builds or generating screenshots. Each of those tools had no connection with each other and ran as standalone commands. About 4 months after the initial release of deliver, snapshot and frameit, I noticed one common theme: All users of those tools developed their own shell scripts to trigger different deployment steps, pass on information, and ensure every step was successful.

I wanted to help developers connect all the tools they use into one simple workflow. That’s why I created fastlane.

But as the community grew and the architecture became more complex, I needed to think about how we could best build the foundation of fastlane to continue to be a tool you love to use. 

In its current setup, we ran into these problems:

  • When releasing a spaceship bugfix, it required us to do at least five releases just to roll out the change to all users
  • fastlane has its own repo just to manage the other repos called “countdown
  • The fastlane organization has over 30 GitHub repositories
  • I added 360 commits whose only purpose was to update internal dependencies between fastlane tools
  • We ended up with a lot of duplicate issues across multiple repos
  • Having 20 different travis builds to keep green, instead of just one
  • Working on a new feature or big change required us to switch to a separate branch on three different repos, which all depended on each other. Not only is it very time consuming to work on three different repo’s feature branches, but also the release takes longer: You have to merge the first PR, then do a release, before you can update the dependency of the other tool, to turn the build green and merge the next one.
  • It’s hard to decide when to release an update for each of the tools, so we built a script to show which tool has the highest number of outstanding changes to be released. Switching to a mono repo allows us to switch to a more regular release cycle

Our goal to unify fastlane tools

To solve a lot of these problems, our team decided to move all fastlane tools into one main repository. Each tool will be in a subfolder enabling you easy access to its source code.

I wanted to make this change as seamless and hassle-free as possible for everyone. With this migration, you can still use all tools directly, so your setup will continue to work as normal. This should not break any existing setups and doesn’t require any action from your side. 

Unifying fastlane means having one place for everything you need. Now, there will be one place to submit and search for issues on GitHub, one place for the documentation, one place to contribute to fastlane, and one tool to install :)

We’ll be making this move within the next couple of weeks.

Technical Details

I’d like to share some more technical information on how the switch will work and the challenges I faced.

The requirements:

  • All individual tools (like deliver, snapshot, etc.) have to be moved into the main fastlane repo
  • We want to preserve the complete history, meaning all commits across all repositories
  • All existing issues on the repositories need to be migrated to the main repo, both open and closed ones
  • I wanted to be sure that every person who contributed to fastlane in the past shows up in the Contribution Graph. fastlane wouldn’t be possible without the great community around it.

To implement all this I created a new repo (ironically) that contains all the scripts used to make the switch. You can find all the source code on fastlane/monorepo.

Moving the tools to the main repo

To move all source code to the main repo and still preserve the history with all commits, we used standard git features. You can check out the full source code on GitHub. Using this technique, all previous commits are still available, and the Contribution Graph shows all developers who ever contributed to fastlane.

Copying the GitHub issues to the main repo

There is no built-in way to migrate or merge issues to another repo. GitHub provides a great API which we used to copy over all existing issues. However the newly copied issues are not created by the original author, but by our fastlane bot instead. On the newly created issue we always mention the original authors, resulting in them being subscribed to the new issue. 
You can check out the full source code on GitHub. With this we worked on a lot of little tricks to make sure users are subscribed to the newly migrated issue and the original author is still visible. The GitHub API doesn’t allow us to post issues as the original author (for good reasons), so we solved this problem by posting the user’s information right inside the actual issue.

Ruby Gems

There are no changes with the way you install fastlane, you can still install tools e.g. using gem install deliver.

What happens to the other GitHub repositories

For SEO reasons we will keep the other repositories around, however the long term plan is to remove them and have everything in the main repo. The old repos will link to the new mono repo, therefore all issues and pull requests will be closed and moved over to the fastlane main repo.

Upgrading fastlane support to the next level 🚀

We took this opportunity to also improve the way we handle issues as a team. Our goal is to be faster and better at responding to all incoming GitHub issues and reviewing pull requests. 

Up until a few months ago, I was managing everything alone, resulting in my not responding to everyone in a timely matter. Thanks to Twitter & Fabric, the fastlane team has grown tremendously in the last few weeks, allowing us to offer world-class help for our thriving community. We want to treat support as a first class citizen and provide the best experience possible.

We’re just getting started: being part of Fabric means a passionate dedicated team to offer the best support & future experience for fastlane customers.

With that in mind, we have a lot of inactive issues and pull requests spanning the last year when I was still working on this project in my free-time. Out of the 829 open issues across all repos, 273 issues haven’t had any activity within the last 3 months. 

After the migration, we will be constantly looking at all issues to make sure the process is improving. We’ll also be commenting on those inactive issues to see if the problem is still happening for those of you who submitted pull requests. This enables us to be much more responsive to new issues and have a better overview on what to focus our time on. 

The Future

This is one of the most impactful changes to fastlane yet. While there’s a lot of information, it’s very important to the entire team to communicate what we are up to. This migration shows our commitment to innovation and creating the future of fastlane .

I’m confident that this change will make yours and our lives much easier and will save us all a ton of time in the future. We’d love your feedback on this, Tweet at us at any time! :) 

Thoughts on iOS build tools

Up until now you had 2 good ways to build and sign your application from the command line:

  • Use the built-in xcodebuild command
  • Use the third party tool shenzhen

Both approaches have their problems

xcodebuild

When using xcodebuild you have to fiddle around with a long list of available parameters:

That's not something you remember and just enter every time you want to build an ipa file. The command above will generate a huge amount terminal output (talking > 10MB). For every source file that gets complied xcodebuild prints out around 60 lines of information you don't really care about:

The best way to fix the output annoyance is to use xcpretty in which you can pipe the xcodebuild ouput.

shenzhen

shenzhen was the easiest way to build your iOS application: It took care of generating the xcodebuild commands to build your application. But instead of showing a clean output while building, it omitted the complete output of the whole process. 

The main issue with shenzhen was the information you received when you run into an error: 

You know what you'd need to do to solve this issue? No, it's not clear, you don't get any information that helps you fix this issue. That's a simple code signing issue, in which the project defines a provisioning profiles that doesn't exist.

shenzhen is a great tool, but hasn't received any updates recently. With fastlane I used a fork of shenzhen to fix some bugs. After a time I decided to build 'gym', a tool which should replace the building part of shenzhen. (shenzhen also does distribution to beta testing services)

Improving the process

In my opinion it's really important a tool does different things depending on the outcome:

  • When the tool succeeded, the user doesn't need a lot of output/information, except that everything worked fine
  • When something went wrong while building, the user should get as much relevant information as possible to make it easy to quickly fix the issue.

Tools should actively help the user resolve common issues. By showing as much useful information as possible and even telling the user how to fix it, a tool automatically becomes much more useful.

Tools can automatically detect all kinds of issues and help the user fix them really fast:

When gym succeeds the terminal output is minimalistic but still shows relevant and useful information while processing

Another important point is being compatible with existing technologies. With gym for example I want the user to still be able to use the Xcode Organiser to distribute the archive or dSYM file

Information

When writing about developer tools I don't want to forget the famous "Fix Issue" button. 

 How can we improve it? 

The most important things a developer tool should do

  • Expose information about what Xcode is going to try when you click "Fix Issue"
  • Show a log of what was tried and what failed
  • If possible show the user how to fix an issue manually

In the example of gym you get every shell command that is being executed printed out so you can manually try fixing something in case the build process fails.

If you need even more information, the you can use --verbose flag.

Conclusion

Besides standard features like sensible defaults and convention over configuration, I think it's important the user knows what's going on with the tool. This allows the users to debug issues themselves. By being transparent you build up trust. Developer see what the tool is doing.

You can check out gym on GitHub: https://github.com/fastlane/gym

Letting computers do the hard work

As iOS developers we're still used to doing many manual tasks. It's an issue that exists because everything is still new. Just ask your back-end developer about the last time they manually deployed a new version directly to the production server. The answer should be: Oh, I was still in high school back then. Why? Because there are many mechanisms before the actual release to avoid broken releases that don't pass the tests.

Using fastlane you can already automate a large part of your daily development tasks, but one thing was missing:

Have you ever been to an airport, where you had to ask the manager of the airport if you can board now? Once the manager agrees, you'll be carried from your check-in to your gate into your plane.

Because that's what you are doing right now as an app developer when you want to invite a beta tester to your TestFlight app. And you even repeat that for every single beta tester you invite.

Right now, you have to go through 9 steps (on the right) just to invite one beta tester to your TestFlight program.

Have you ever asked a blog publisher if they add you to their newsletter? No, that's not how that works!

I just launched boarding, a tool that allows you to launch your own TestFlight Invite page in under 3 minutes.

More information about this project can be found on GitHub.

About Automation

If you're reading this post, chances are high you are an iOS/Mac developer. You get paid for developing iOS applications which is probably what you're best in.

Are you responsible for...?

  • Creating and uploading screenshots
  • Updating app metadata like the description
  • Manually building and uploading a beta build when your boss tells you to
  • Inviting new beta testers to your beta testing service

You probably said yes to some of those points. In my experience most developers do those things manually. Why?

We haven't got time to automate this stuff, because we're too busy dealing with the problems caused by our lack of automation. - @bitfield

While some developer may enjoy doing passive activities, you usually want to get your job done. That means working on the iOS/Mac app itself working on awesome new features.

That's why we have to follow our friends from the back-end team and start automating tedious processes. Instead of us, computer should do those tasks.

As you can see, the users interested in beta testing your application accesses your Heroku application directly to enter their email address. The Heroku app will then use spaceship to communicate with iTunes Connect directly without any interaction from a real person. Spaceship automatically registers the new testers, adds them to the application and sends out the TestFlight email. 

You as an developer are in no way involved in this process. Once you merge into your beta branch, fastlane will go ahead and build, sign and upload your app, resulting in automatic emails to all your testers. (this depends on how you use fastlane)

If you tried the Heroku button, you'll see there are no additional manual steps required to get a web service running. You enter your credentials, choose your application and the web service is up and running. 

What's next?

The primary use case of boarding was not to actually solve this certain problem, but to demonstrate what's possible using spaceship. In only 24 hours I had the idea for boarding, started working on it and released it publicly.

Think about what you can build with spaceship. It's a foundational tool to communicate with Apple's web services, both the Apple Developer Portal and iTunes Connect allowing you to build all kinds of cool things!

Some random ideas:

  • Automatically sync iTunes Connect users with your company internal user system
  • Automatic daily backups of all iTunes Connect and Apple Dev Portal data
  • Automatically send customers emails once a new version of their app got approved
  • Build a lightweight dashboard for your clients to see a grid of their apps with the current download numbers, reviews and app review status
  • Using the same change-log text for all languages? Why should you copy&paste the text manually to 10 languages?
  • Added App Review information to your company dashboard or push them directly to Slack

Managing fastlane feedback

I'm happy to say, most feedback I receive for the fastlane projects is very positive. I think that's mostly because it's something that's completely new in the iOS field and people just realised: Hey, we can actually automate those kind of things.

Sometimes we forget the happiest moments. Remember the time your role model tweeted about you or one of your projects? Remember the time a well known developer retweeted your last tweet? It's the best thing, you are insanely happy for a while, but forget about it again a few days later. 

But that doesn't only apply to tweets: How about the last time one of your projects became a trending repository on GitHub? What about that email you received from a fan who just wanted to say Thanks?

It might be just be me, but I don't want to lose that "moments". I even collected screenshots of the "follow" notifications I received when some of my favourite developers like @orta, @tapbot_paul ,@ashfurrow, @javi@soffes and @kylefuller started following me on Twitter.

I just didn't want to lose all this. That's why I collected the best tweets and screenshots since the initial launch of deliver

Open All Tweets

Screenshots

Besides tweets, you probably want to remember all kinds of things: comments on Reddit, the number of visitors you had on launch day and smaller things like the number of git commits you had on that one day.

Emails

I'm still using Sparrow, the best mail client for the Mac. To store an email, I just drag and drop the email into the fastlane feedback folder.

And now what?

So why do all this? I'm not a super organised person, but I try my best to do this right. After storing all this, I usually print it all and store it somewhere save. All this just because you never know what formats will be readable in 20 years and if you "lose" some files because you forgot to migrate some of those files. 

Check out the full list of tweets