Writing automated tests for your documentation

The 2 favourite topics of every engineer are usually unit tests and documentation. The last few weeks I've put some thoughts in how to combine those two things for the ultimate developer joy.

Why should I write tests for my documentation? 

  • You might forget to update parts of the documentation when you update your API or interface
  • Spelling mistakes happen in code samples
  • You want to make sure you don't introduce any regressions with a new release

With the launch of the new docs.fastlane.tools website, we took the time to set up a proper documentation system based on markdown files, converted to a static HTML page using MkDocs. By having full control over the generated HTML code, and going 100% open source on GitHub, you can do powerful things:

  • Fail the build when a page got removed from the index using danger (see Dangerfile)
  • Generate redirects when a page got moved (see generate_redirects.rb and available_redirects.rb)
  • Make sure the repo is in a valid states, and all documentation follows the convention
  • Write tests for your documentation

Tests for documentation, what does this even mean?

Just like with your actual code base, you want to be sure that your repository is in a valid state. The same applies for your documentation. Is your latest documentation up to date, and does it work as expected? Does it work with your latest release?

The first time I thought about running tests for documentation I thought only about the code samples. However you can write tests for a lot of more things. For example you want to ensure that all your assets are in the right sub-directory, that all referenced images and markdown files are available and follow your naming scheme.

Some examples of what you can do:

Validate file names

Ensure that all file names follow your convention - this is especially important if the file names are visible in the URL.

Ensure every page has a header

fastlane docs require a title on each page to be consistent.

Ensure all image assets are available and in the correct folder

Automatically verify all referenced image assets are available and stored in the right folder, find the source here.

Ensure spelling of certain words

Recently we switched from `fastlane` to _fastlane_ in markdown files, so we want to ensure all new docs use the new way of spelling.

Verify code samples

How many times have you copied a code sample to try it out, just to find out it doesn't work with the latest release? There are various reasons for this, and without tests there is almost no way to avoid this. The error can be as simple as a spelling mistake.

By having full control over your documentation, you can easily iterate over all your code samples, and make sure they work as expected.

For fastlane we found 28 errors just in the actions documentation, most commonly syntax errors or just simple spelling mistakes in parameter names.

For all the checks mentioned below, the source is on the fastlane docs repo.

Catch syntax errors in code samples

Nothing is more frustrating than copying a code sample and having to fix the syntax to make it work. Most users won't spend the time to look into how they can contribute to your docs to fix the error.

 

 

Catch unavailable actions or methods

For fastlane most code samples call certain actions or integrations. Using tests we ensure that this action is available and correctly spelled.

 

Verify action parameters

Most fastlane actions allow passing of named parameters. We now automatically verify that all parameters are available to use.

 

Verify parameter types

Every fastlane parameter has a type defined, which fastlane validates on run-time. It's easy to get a type wrong in the documentation, so we now validate those too.

 

 

When to run tests for your docs?

There are 2 points where you want to verify your docs are valid and up to date:

So whatever introduced a regression, the build will be failing and you can't merge the change into master.

Where to go next

These are just some things you can do with tests for your documentation, however I'm sure there are so many things you can automatically test and verify every time your code changes. Let me know if you have any ideas on what other crazy things you can do.

'trainer' - the simplest way to generate a JUnit report of your iOS tests

If you're running tests for your iOS code base on some kind of Continuous Integration, you usually have to generate a JUnit file to report the test results, including error information, to your CI system. Until now, the easiest solution was to use the amazing xcpretty, which parses the xcodebuild output and converts it to something more readable, additionally to generating the JUnit report.

Since there were some difficulties with the new Xcode 8, Peter Steinberger and me had the idea to parse the test results the same way Xcode server does it: using the Xcode plist logs

trainer is a simple standalone tool (that also contains a fastlane plugin), which does exactly that: Convert the plist files to JUnit reports.

By using trainer, the Twitter iOS code base now generates JUnit reports 10 times faster.

To start using trainer, just add the following to your Fastfile:

By combining trainer with danger, you can automatically show the failed tests right in your pull request, without having to open the actual output

For more information about when to use trainer, check out the full blog article on PSPDFKit.

Introducing fastlane device grid, one-click installation, 2FA, more APK management features, and emoji deployment

In May, we focused on making pull requests more powerful and introduced enhanced deployment options. For more details, check out the full blog post on fabric.io/blog.

 

fastlane device grid

Tired of locally compiling your app just to test and preview a change? From now on, you can test your app straight on GitHub from the pull request using device grid.

One-click installation via the Fabric Mac app

Want to install fastlane with one quick click? You can do it directly from the Fabric Mac app. 

Two-step verification

Looking to take advantage of two-step verification for stronger security? All fastlane tools now support two factor authentication. 

APK expansion file upload and APK promote to track capability

Last but not least, thanks to awesome contributions from the community, it’s easier than ever to upload APK expansion files and promote existing beta builds to production on Google Play. 

Read the full blog post on fabric.io/blog

Be a 10x Mac user by using custom shortcuts for every application you use

One of the most common activities when using a computer is switching between applications and their windows. Unfortunately Mac OS X is optimised for casual computer users (Mission Control, using the Mouse, etc.).

Using the dock

This approach requires the mouse and wastes either screen space, or has a slow animation when using auto-hide.

Using CMD + Tab

 
 

This approach is keyboard based, however the order of the application is sorted by most recently used, which is only nice to quickly switch between 2 applications.

Additionally this technique doesn't allow you to launch a new application, but only switch between running ones.

Using Spotlight or Alfred

 
 

Great to quickly launch any application, however it's not as fast as other approaches, as it requires you to type in multiple letters.

 

New: Launch every application using a shortcut

When people watch me use my computer I get asked a lot how I can switch between my applications so quickly.

I'm not kidding, I have a custom keyboard shortcut for every single application I use frequently.

This way I can always switch to any application just using one keyboard shortcut. 

I use the excellent tool called BetterTouchTool, which was originally built to use custom gestures for the trackpad and mouse, however I use it mostly for my shortcuts. 

Using BetterTouchTool, it will either switch to a running application or launch it otherwise.

Some tricks:

  • ALT + [letter]: Open a specific application (e.g. ALT + n launches Notes)
  • Alt + 2: Open the screenshots directory (use this for any directory you use often)
  • Ctrl + CMD + Arrow: Move window to next monitor

Do you have other ideas for useful shortcuts? Please feel free to add them as a comment

Automatically download and upload dSYM symbolication files from iTunes Connect for Bitcode iOS apps using fastlane

When submitting iOS apps to the App Store with Bitcode enabled, your app gets recompiled by Apple, to be optimized for specific devices and architectures. While Bitcode is optional, it's more and more encouraged by Apple, and even required for watchOS and tvOS apps.

Due to the fact that recompiling happens on Apple's servers, third party crash reporters don't have access to the symbolication files, which are required to properly associate the crash stack trace with the exact files and line numbers in your source code.

The current solution to solve this for Bitcode enabled apps is to manually download the dSYM files from Apple and upload them to your crash reporting service. Downloading the files can be done either using Xcode or iTunes Connect. 

Download dSYM symbolication files using the Xcode Organizer

Download dSYM symbolication files from iTunes Connect

The problem with this approach is not only the time you spend doing this manually for every single release, but also the lack of automation. There was no way to automatically do this process after every release, or just periodically on every day, so you end up spending a lot of engineering time on repetitive tasks.

Introducing automatic fastlane dSYM download

With the latest version of fastlane, you can now easily download all available dSYM symbolication files from iTunes Connect and upload them straight to the crash reporting service of your choice.

All you have to do is define a lane (as seen above) and run:

           $ fastlane refresh_dsyms

It's easy to just run this task on your existing CI infrastructure every day or so, however you're totally free to trigger it whenever and wherever you want. 

dSYM download in action - click on the gif to start it from the beginning

You can find more information about the available options in the fastlane docs.

Currently the dSYM upload is supported by Crashlytics, Sentry and HockeyApp. If your service is still missing and they provide an API, feel free to submit a PR to add it to fastlane.

Open Source

As always, everything described in this article is completely open source under the MIT license, check out the following source files:


Getting started

If you're already using fastlane, just update your existing Fastfile and add the new refresh_dsyms lane to it.

Step 1: Install fastlane

If you haven't yet had the change to try fastlane, now is the perfect time. To quickly get started:

           $ [sudo] gem install fastlane --verbose
           $ cd your/project
           $ mkdir fastlane && touch fastlane/Fastfile fastlane/Appfile

Step 2: Update Appfile

Edit your Appfile to include your bundle identifier and iTunes Connect Apple ID.

Step 3: Update Fastfile

Update your Fastfile to include the refresh_dsyms lane (seen above). You're all set, now you can run 

Step 4: Enjoy fastlane

           $ fastlane refresh_dsyms

On the first run you'll be asked for your password, which will be stored in your local Keychain. 

You wonder what else you can add to your Fastfile? Check out the available actions to see all 170 built-in fastlane integrations, you can already use today. 

Our goal to unify fastlane tools

I’ve been working as an iOS developer for over 5 years for various companies around the world. During that time it became clear there was lots of room for improvements around mobile developer tools, and in particular, the deployment of apps. Because of that, I started implementing little tools to help me automate tedious tasks like uploading metadata for release builds or generating screenshots. Each of those tools had no connection with each other and ran as standalone commands. About 4 months after the initial release of deliver, snapshot and frameit, I noticed one common theme: All users of those tools developed their own shell scripts to trigger different deployment steps, pass on information, and ensure every step was successful.

I wanted to help developers connect all the tools they use into one simple workflow. That’s why I created fastlane.

But as the community grew and the architecture became more complex, I needed to think about how we could best build the foundation of fastlane to continue to be a tool you love to use. 

In its current setup, we ran into these problems:

  • When releasing a spaceship bugfix, it required us to do at least five releases just to roll out the change to all users
  • fastlane has its own repo just to manage the other repos called “countdown
  • The fastlane organization has over 30 GitHub repositories
  • I added 360 commits whose only purpose was to update internal dependencies between fastlane tools
  • We ended up with a lot of duplicate issues across multiple repos
  • Having 20 different travis builds to keep green, instead of just one
  • Working on a new feature or big change required us to switch to a separate branch on three different repos, which all depended on each other. Not only is it very time consuming to work on three different repo’s feature branches, but also the release takes longer: You have to merge the first PR, then do a release, before you can update the dependency of the other tool, to turn the build green and merge the next one.
  • It’s hard to decide when to release an update for each of the tools, so we built a script to show which tool has the highest number of outstanding changes to be released. Switching to a mono repo allows us to switch to a more regular release cycle

Our goal to unify fastlane tools

To solve a lot of these problems, our team decided to move all fastlane tools into one main repository. Each tool will be in a subfolder enabling you easy access to its source code.

I wanted to make this change as seamless and hassle-free as possible for everyone. With this migration, you can still use all tools directly, so your setup will continue to work as normal. This should not break any existing setups and doesn’t require any action from your side. 

Unifying fastlane means having one place for everything you need. Now, there will be one place to submit and search for issues on GitHub, one place for the documentation, one place to contribute to fastlane, and one tool to install :)

We’ll be making this move within the next couple of weeks.

Technical Details

I’d like to share some more technical information on how the switch will work and the challenges I faced.

The requirements:

  • All individual tools (like deliver, snapshot, etc.) have to be moved into the main fastlane repo
  • We want to preserve the complete history, meaning all commits across all repositories
  • All existing issues on the repositories need to be migrated to the main repo, both open and closed ones
  • I wanted to be sure that every person who contributed to fastlane in the past shows up in the Contribution Graph. fastlane wouldn’t be possible without the great community around it.

To implement all this I created a new repo (ironically) that contains all the scripts used to make the switch. You can find all the source code on fastlane/monorepo.

Moving the tools to the main repo

To move all source code to the main repo and still preserve the history with all commits, we used standard git features. You can check out the full source code on GitHub. Using this technique, all previous commits are still available, and the Contribution Graph shows all developers who ever contributed to fastlane.

Copying the GitHub issues to the main repo

There is no built-in way to migrate or merge issues to another repo. GitHub provides a great API which we used to copy over all existing issues. However the newly copied issues are not created by the original author, but by our fastlane bot instead. On the newly created issue we always mention the original authors, resulting in them being subscribed to the new issue. 
You can check out the full source code on GitHub. With this we worked on a lot of little tricks to make sure users are subscribed to the newly migrated issue and the original author is still visible. The GitHub API doesn’t allow us to post issues as the original author (for good reasons), so we solved this problem by posting the user’s information right inside the actual issue.

Ruby Gems

There are no changes with the way you install fastlane, you can still install tools e.g. using gem install deliver.

What happens to the other GitHub repositories

For SEO reasons we will keep the other repositories around, however the long term plan is to remove them and have everything in the main repo. The old repos will link to the new mono repo, therefore all issues and pull requests will be closed and moved over to the fastlane main repo.

Upgrading fastlane support to the next level 🚀

We took this opportunity to also improve the way we handle issues as a team. Our goal is to be faster and better at responding to all incoming GitHub issues and reviewing pull requests. 

Up until a few months ago, I was managing everything alone, resulting in my not responding to everyone in a timely matter. Thanks to Twitter & Fabric, the fastlane team has grown tremendously in the last few weeks, allowing us to offer world-class help for our thriving community. We want to treat support as a first class citizen and provide the best experience possible.

We’re just getting started: being part of Fabric means a passionate dedicated team to offer the best support & future experience for fastlane customers.

With that in mind, we have a lot of inactive issues and pull requests spanning the last year when I was still working on this project in my free-time. Out of the 829 open issues across all repos, 273 issues haven’t had any activity within the last 3 months. 

After the migration, we will be constantly looking at all issues to make sure the process is improving. We’ll also be commenting on those inactive issues to see if the problem is still happening for those of you who submitted pull requests. This enables us to be much more responsive to new issues and have a better overview on what to focus our time on. 

The Future

This is one of the most impactful changes to fastlane yet. While there’s a lot of information, it’s very important to the entire team to communicate what we are up to. This migration shows our commitment to innovation and creating the future of fastlane .

I’m confident that this change will make yours and our lives much easier and will save us all a ton of time in the future. We’d love your feedback on this, Tweet at us at any time! :)