Writing automated tests for your documentation

The 2 favourite topics of every engineer are usually unit tests and documentation. The last few weeks I’ve put some thoughts in how to combine those two things for the ultimate developer joy.

Why should I write tests for my documentation?

You might forget to update parts of the documentation when you update your API or interface
Spelling mistakes happen in code samples
You want to make sure you don’t introduce any regressions with a new release

With the launch of the new docs.fastlane.tools website, we took the time to set up a proper documentation system based on markdown files, converted to a static HTML page using MkDocs. By having full control over the generated HTML code, and going 100% open source on GitHub, you can do powerful things:

Fail the build when a page got removed from the index using danger (see Dangerfile)
Generate redirects when a page got moved (see generate_redirects.rb and available_redirects.rb)
Make sure the repo is in a valid states, and all documentation follows the convention
Write tests for your documentation

Tests for documentation, what does this even mean?

Just like with your actual code base, you want to be sure that your repository is in a valid state. The same applies for your documentation. Is your latest documentation up to date, and does it work as expected? Does it work with your latest release?

The first time I thought about running tests for documentation I thought only about the code samples. However you can write tests for a lot of more things. For example you want to ensure that all your assets are in the right sub-directory, that all referenced images and markdown files are available and follow your naming scheme.

Some examples of what you can do:

Validate file names

Ensure that all file names follow your convention - this is especially important if the file names are visible in the URL.

private_lane :ensure_filenames do
  UI.message "🕗  Making sure file names are all lowercase..."
  Dir["docs/**/*.md"].each do |current_file|
    if current_file =~ /[A-Z]/
      UI.user_error!(".md files must not be upper case, use a dash instead (file '#{current_file}')")
    end
  end
  UI.success "✅  All doc files are properly named to not be camel case"
end

Ensure every page has a header

fastlane docs require a title on each page to be consistent.

private_lane :ensure_headers do
  UI.message "🕗  Investigating all docs files and their headers..."
  exceptions = ["docs/index.md"]

  Dir["docs/**/*.md"].each do |current_file|
    next if exceptions.include?(current_file)
    unless File.read(current_file).start_with?("#"))
      UI.user_error!("File '#{current_file}' doesn't start with #"
    end
  end
  UI.success "✅  All doc files start with a header"
end

Ensure all image assets are available and in the correct folder

Automatically verify all referenced image assets are available and stored in the right folder, find the source here.

Ensure spelling of certain words

Recently we switched from fastlane to fastlane in markdown files, so we want to ensure all new docs use the new way of spelling.

private_lane :ensure_tool_name_formatting do
  UI.message "🕗  Verifying tool name formatting..."
  require 'fastlane/tools'

  Dir["docs/**/*.md"].each do |path|
    content = File.read(path)
    Fastlane::TOOLS.each do |tool|
      if content.include?("`#{tool}`")
        UI.user_error!("Use _#{tool}_ instead of `#{tool}` to mention a tool in the docs in '#{path}'")
      end

      if content.include?("_#{tool.to_s.capitalize}_") || content.include?("`#{tool.to_s.capitalize}`")
        UI.user_error!("fastlane tools have to be formatted in lower case: #{tool} in '#{path}'")
      end
    end
  end
  UI.success "✅  fastlane formatting is correct"
end

Verify code samples

How many times have you copied a code sample to try it out, just to find out it doesn’t work with the latest release? There are various reasons for this, and without tests there is almost no way to avoid this. The error can be as simple as a spelling mistake.

By having full control over your documentation, you can easily iterate over all your code samples, and make sure they work as expected.

For fastlane we found 28 errors just in the actions documentation, most commonly syntax errors or just simple spelling mistakes in parameter names.

For all the checks mentioned below, the source is on the fastlane docs repo.

Catch syntax errors in code samples

Nothing is more frustrating than copying a code sample and having to fix the syntax to make it work. Most users won’t spend the time to look into how they can contribute to your docs to fix the error.

Catch unavailable actions or methods

For fastlane most code samples call certain actions or integrations. Using tests we ensure that this action is available and correctly spelled.

Verify action parameters

Most fastlane actions allow passing of named parameters. We now automatically verify that all parameters are available to use.

Verify parameter types

Every fastlane parameter has a type defined, which fastlane validates on run-time. It’s easy to get a type wrong in the documentation, so we now validate those too.

When to run tests for your docs?

There are 2 points where you want to verify your docs are valid and up to date:

When you push a new release of your software (deploy script of your software)
When you update your documentation (CI test script of your docs)

So whatever introduced a regression, the build will be failing and you can’t merge the change into master.

Where to go next

These are just some things you can do with tests for your documentation, however I’m sure there are so many things you can automatically test and verify every time your code changes. Let me know if you have any ideas on what other crazy things you can do.

Felix Krause