Great Rust CI

Great CI means coding with confidence. And the CI-compatible tooling situation for Rust is great, so you should take advantage of it.

If you don't know the full details of all of the available tools, though, it can be difficult to set up the perfect CI. So here, I'll walk you through the setup that I'm using.

Install local tools.

Here's what I use every day:

1. Clippy: code linting for more opinionated things than what rustc gives you.

   For now, clippy will only work with the latest nightly. Install and update nightly rust with rustup, then run cargo +nightly install clippy --force to force a clippy install built against your current nightly. Use it with cargo +nightly clippy.

2. Rustfmt: automatic code formatting.

   For many (though not all, sadly) nightlies, rustfmt can be managed using rustup, just like you can for rls. If the nightly that you're on contains rustfmt, you can do rustup component add rustfmt-preview. If your nightly doesn't contain rustfmt-preview, you can install it from crates.io: cargo +nightly install rustfmt-nightly --force. Having both the rustup managed and cargo managed versions can lead to conflicts, so use one or the other. Use it with cargo +nightly fmt.

3. Cargo-Update: check for and update cargo-install-ed binaries.

   Not as much help for clippy/rustfmt, which need to be force-reinstalled for every new nightly (because they link against it), but invaluable for seamlessly updating other tools (including itself). (Requires CMake)

4. Optional: Cargo-Edit: modify the build manifest (Cargo.toml) from the command line with simple commands, rather than editing the file by hand.

5. Optional: IDE support. I use IntelliJ IDEA with the IntelliJ Rust plugin. There's also RLS for VSCode, or you could use Vim or Emacs if you're into that kind of thing (no judging!).

Create your project!

$ cargo new great-ci
     Created library `great-ci` project

It's good practice to set up .git/info/exclude to tell git to ignore your IDE files. For me, that means the contents of the git exclude are:

.idea/
*.iml

And for the purpose of this example, let's make a tiny "Hello World" style library:

//! Example library for great CI integration!

use std::fmt;

/// Greetings to some target
#[derive(Copy, Clone, Debug, Eq, PartialEq, Hash)]
pub struct Greeting<'a>(&'a str);

impl<'a> Greeting<'a> {
    /// Construct a Greeter to greet a target
    pub fn greet(target: &str) -> Greeting {
        Greeting(target)
    }
}

impl<'a> fmt::Display for Greeting<'a> {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "Greetings, {}!", self.0)
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn it_works() {
        assert_eq!(2 + 2, 4);
    }

    #[test]
    fn greet_author() {
        let greeter = Greeting::greet("CAD97");
        assert_eq!(greeter.to_string(), "Greetings, CAD97!");
    }
}

Add a LICENSE and README, then it's time to commit and get the code online.

$ git add .
$ git commit -m "Initial commit"
$ git remote add CAD97 git@github.com:CAD97/great-ci.git
$ git push -u CAD97 master

Set up CI

The part that we care about! I use Travis CI, Cargo-Travis, Codecov, and Bors for my CI stack.

So let's switch to a feature branch and set up a basic bors.toml

status = [
    "continuous-integration/travis-ci/push"
]

and .travis.yml:

language: rust

rust:
  - stable
  - beta
  - nightly

script: |
  cargo build --verbose &&
  cargo test  --verbose &&
  cargo doc   --verbose

branches:
  only:
    - staging # bors r+
    - trying  # bors try
    - master

don't forget to enable Travis and Bors for the repository (I did), and then send a PR. If all goes well, your CI is green and you can merge with bors r+.

Testing rustfmt and clippy

When you're working with a project you expect to get large, it can help to enforce standard formatting, and clippy does wonders for preventing simple mistakes and nudging people away from questionable API decisions.

Because rustfmt and clippy are somewhat unstable, requiring certain small ranges of nightly compilers and sometimes breaking, and changing default formatting or adding lints may break your build, we depend on specific versions of the tools, and you can update the versions manually.

We use a matrix include on Travis to add the check:

cache: cargo
matrix:
  include:
    - rust: nightly-2018-01-12
      env: # use env so updating versions causes cache invalidation
        - CLIPPY_VERSION=0.0.179
      before_script:
        - rustup component add rustfmt-preview
        - cargo install clippy --version $CLIPPY_VERSION || echo "clippy already installed"
      script:
        - cargo fmt -- --write-mode=diff
        - cargo clippy -- -D clippy

Here we install clippy using cargo install, or succeed if it's already installed. We install rustfmt using rustup, because in Travis's Rust setup, rustfmt is already tracked by rustup, so the cargo-fmt executable exists, and an install without --force will fail. And if we don't pick a nightly toolchain that actually contains cargo-fmt, using it will fail. Here we have rust: nightly-2018-01-12 CLIPPY_VERSION=0.0.179 because this is the latest toolchain as-of-writing where rustfmt-preview is in, and the latest version of clippy which builds on that nightly.

Installing tools like rustfmt and clippy takes time, and CI is made great when it's fast, so we enable Travis's Rust Cargo cache to cache cargo's symbols. This speeds up the build because cargo doesn't have to recompile dependencies, whether these be build dependencies or tools.

The cache is keyed by the language version (here rust: nightly-2018-01-18) and the env (here CLIPPY_VERSION=0.0.180), and shared between build jobs with the same rust version and env. By putting clippy's version in the env, we separate the tool build's cache from the other caches, and ensure that a version update causes a rebuild.

You may be tempted to stick these environment variables in the CI configuration, so that you can update them without an extra commit to the repository. This is ill-advisable, for two main reasons. Primarily, version bumping these tools has a high likelihood of changing requirements for your build, and the breakage will show up unannounced. This change should be marked in your repository. Secondarily, the use of env here also serves to separate the cache for these tools from the other caches. If the specified rust version is the most up-to-date nightly, this cache and the nightly cache would clobber each other. And you don't want to force installation of these tools on CI builds that don't require it, because we want those zippy CI greens.

So make these changes, submit a PR, and we're go for the last step!

Using cargo-travis

Cargo-travis enables you to easily use kcov and upload coverage information to codecov or coveralls. Additionally, added by myself recently, it allows you to upload documentation for your crate to GitHub pages and maintain an understandable git history and directory structure allowing you to document multiple branches (master/beta/release branches to imitate Rust's nightly/beta/stable trains) if you so wish.

Because these tasks can take a while, and are more for informing decisions rather than to hard gate PRs, I stick these into a allowed-failure matrix line for my builds. This means that Travis passes as soon as the four above jobs pass (rust: [stable, beta, nightly], rustfmt&clippy), and allows the cargo-travis job to continue running in the background.

Our .travis.yml additions:

env: # required for allow_failures
matrix:
  fast_finish: true
  allow_failures:
    - env: NAME='cargo-travis'
  include:
    - env: NAME='cargo-travis'
      sudo: required # travis-ci/travis-ci#9061
      before_script:
        - cargo install cargo-update || echo "cargo-update already installed"
        - cargo install cargo-travis || echo "cargo-travis already installed"
        - cargo install-update -a
      script:
        - |
          cargo build    --verbose &&
          cargo coverage --verbose &&
          bash <(curl -s https://codecov.io/bash) -s target/kcov
        - |
          cargo doc --verbose &&
          cargo doc-upload
      addons: # required for kcov
        apt:
          packages:
            - libcurl4-openssl-dev
            - libelf-dev
            - libdw-dev
            - binutils-dev
            - cmake

If you want to use Coveralls, just use cargo coveralls instead of cargo coverage. If you don't want to use Codecov, just remove the line for the Codecov bash script.

This isn't enough for the doc-upload task, however (though this will not make your CI go red, as we allow this job to fail). In order to upload documentation, the doc-upload script needs permission to push to your GitHub repository. The easiest way, which we will use here, is a GitHub token, but you can read more about the options at the cargo-travis README.

In order to generate a GitHub token, navigate to your tokens settings page. While you're there, review your currently issued tokens, and revoke those that you're no longer using. Generate a new token, give it a descriptive name (this can be edited later) (I suggest repo-name-doc-upload) and the public_repo scope (this can be edited later). Make sure to copy the token once it's generated, as you will never be able to see it again (though you can regenerate it)! This token gives anyone with the token access to act as you when read/writing to any public repository you have the permission to read/write to, so keep it safe. Add it to the GH_TOKEN environment variable on Travis and make sure that "Display value in build log" is set to off.

Once you send your PR, the new job will kick off.

A few things to note about this setup:

• CI will pass before coverage is done. This is intentional, as we want to see quick CI feedback for tests passing/failing, and coverage doesn't need to slow down that feedback loop for PRs (or the time it takes to bors r+ and merge). We still want to see those stats, though, and Codecov/Coveralls will still do its analysis and leave a comment once the coverage information is uploaded.
• cargo-travis will take a good long time the first time around; you have to build a large number of tools, including kcov itself. Hopefully, however, all of this will be cached, so subsequent builds will be much faster.
• Documentation will be built when CI builds the master branch only. You can specify which branches to build by passing (optionally multiple) --branch NAME arguments to cargo doc-upload.
• doc-upload does not build documentation for you, so you need to call cargo doc yourself. This is so that you can, if your library offers feature flags or other conditional compilation, build up a directory in target/doc (rustdoc's output directory) which doc-upload will then use when uploading to GitHub pages.
• Documentation ends up living at https://<user>.github.io/<repo>/<branch>/<crate>/, which is https://cad97.github.io/great-ci/master/great_ci/ for this example.
• Coverage is 100% 🎉

Further extensions

If you want to (or need to) test on a Windows platform, AppVeyor is often used. This CI can be used in parallel to this Travis setup with no problems. Make sure to tell Bors to wait for the AppVeyor build, however!

Don't forget that you can suppress rustfmt and clippy warnings if you don't want them to apply to certain blocks of code. If it's an issue in the source library, submit an issue, or if it's just a disagreement, check the configuration and see if you can change it there.

For the security-minded, the public_repo scope of the GitHub token may be too broad. This is a solved problem; the solution is repo-specific deploy keys. If you provide a deploy key for your repository and load it into Travis (and don't provide a token), doc-upload will use it.

You've got the CI set up, brag about it in your README using badges and by embedding coverage charts! Shields|IO provides consistently-styled badges for practically every service, and Codecov graphs are 🔥🔥🔥.

The base page of your GitHub pages (<user>.github.io/<repo>/) is going to 404, as no index.html exists at the root of your gh-pages branch. The same for the branch root (<user>.gihub.io/<repo>/<branch>). doc-upload deliberately doesn't mess with your root directory or the index.html at the branch root so that you can include a HTML redirect at these spots or a more informative index if you want. For a HTML redirect, use the following:

<meta http-equiv="refresh" content="0; url=<crate>">
<a href="<crate>">Redirect</a>

Discuss this on Reddit and tell me if I left something out or something is outdated!

See the public repository on GitHub!