Writing automated tests for your documentation

The 2 favourite topics of every engineer are usually unit tests and documentation. The last few weeks I've put some thoughts in how to combine those two things for the ultimate developer joy.

Why should I write tests for my documentation? 

  • You might forget to update parts of the documentation when you update your API or interface
  • Spelling mistakes happen in code samples
  • You want to make sure you don't introduce any regressions with a new release

With the launch of the new docs.fastlane.tools website, we took the time to set up a proper documentation system based on markdown files, converted to a static HTML page using MkDocs. By having full control over the generated HTML code, and going 100% open source on GitHub, you can do powerful things:

  • Fail the build when a page got removed from the index using danger (see Dangerfile)
  • Generate redirects when a page got moved (see generate_redirects.rb and available_redirects.rb)
  • Make sure the repo is in a valid states, and all documentation follows the convention
  • Write tests for your documentation

Tests for documentation, what does this even mean?

Just like with your actual code base, you want to be sure that your repository is in a valid state. The same applies for your documentation. Is your latest documentation up to date, and does it work as expected? Does it work with your latest release?

The first time I thought about running tests for documentation I thought only about the code samples. However you can write tests for a lot of more things. For example you want to ensure that all your assets are in the right sub-directory, that all referenced images and markdown files are available and follow your naming scheme.

Some examples of what you can do:

Validate file names

Ensure that all file names follow your convention - this is especially important if the file names are visible in the URL.

Ensure every page has a header

fastlane docs require a title on each page to be consistent.

Ensure all image assets are available and in the correct folder

Automatically verify all referenced image assets are available and stored in the right folder, find the source here.

Ensure spelling of certain words

Recently we switched from `fastlane` to _fastlane_ in markdown files, so we want to ensure all new docs use the new way of spelling.

Verify code samples

How many times have you copied a code sample to try it out, just to find out it doesn't work with the latest release? There are various reasons for this, and without tests there is almost no way to avoid this. The error can be as simple as a spelling mistake.

By having full control over your documentation, you can easily iterate over all your code samples, and make sure they work as expected.

For fastlane we found 28 errors just in the actions documentation, most commonly syntax errors or just simple spelling mistakes in parameter names.

For all the checks mentioned below, the source is on the fastlane docs repo.

Catch syntax errors in code samples

Nothing is more frustrating than copying a code sample and having to fix the syntax to make it work. Most users won't spend the time to look into how they can contribute to your docs to fix the error.

 

 

Catch unavailable actions or methods

For fastlane most code samples call certain actions or integrations. Using tests we ensure that this action is available and correctly spelled.

 

Verify action parameters

Most fastlane actions allow passing of named parameters. We now automatically verify that all parameters are available to use.

 

Verify parameter types

Every fastlane parameter has a type defined, which fastlane validates on run-time. It's easy to get a type wrong in the documentation, so we now validate those too.

 

 

When to run tests for your docs?

There are 2 points where you want to verify your docs are valid and up to date:

So whatever introduced a regression, the build will be failing and you can't merge the change into master.

Where to go next

These are just some things you can do with tests for your documentation, however I'm sure there are so many things you can automatically test and verify every time your code changes. Let me know if you have any ideas on what other crazy things you can do.