Learn how to test your docs in CI/CD with Runme's latest v1.0 release, which comes with new cool features you can utilize to deliver seamless developer experience. We will leverage Runme's new kernel architecture enabling seamless interoperability between notebook and terminal UX.

⏯ Never heard of Runme before? It's an open source toolkit that let's you run your README.md and other markdown docs in both terminal and as runnable notebook inside VS Code. Check it out.

Integration testing docs in GitHub Actions

Let's See How

Let's jump right into a showcase. A “thank you” goes out to our friends at Buoyant (creators of Linkerd, the open-source service mesh), who openly maintain Linkerd’s docs in Markdown. That same Markdown generates static HTML on their website for developers to copy commands into the terminals. It turns out that Linkerd’s Getting Started Guide makes a great real-world example illustrating how to harness Runme in CI/CD (with minimal to no edits).

Markdown docs are everywhere describing Developer Experience

The End-to-End Demo

While you could watch the GitHub Action running in real-time, we can’t guarantee that the GitHub Action is running at all times, so we created a video. The time-lapse video (eventual consistency in Kubernetes takes time) illustrates side-by-side what’s going on when testing Linkerd’s Getting Started guide and Kubernetes cluster state. This exact setup will be executed inside a CI/CD job.

Full end-to-end demo as time-lapse video

Inside of GitHub Actions

Once packaged up in a Github Action workflow, running on every commit is just a matter of configuration. Go ahead and send a PR to follow along in real time! It’s all public:

Live at https://github.com/stateful/linkerd-website/actions/workflows/dx.yml

For the curious ones, you can find the details of how we wired up the GitHub Action’s workflow with GCP and Runme inside the repo. The hard part’s done. Now, in about 5min after a commit, you will have assurance (watch out for that 🟢 light!) whether or not the Getting Started Guide still works and your Developer Experience remains intact.

Breaking It Down

Before getting into improvements, let’s take a closer look at the moving pieces of this demo. Linkerd (version 2) is a Kubernetes-native service mesh. We will need a functioning cluster to test the Getting Started Guide via the Linkerd CLI. There are many permutations of CI/CD and cluster setups/provider combos. We won’t be covering them.

Tools and Infrastructure

Instead, we’ve chosen well-known and established ingredients to keep it simple. The setup:

• Standard Kubernetes Cluster in Google’s GKE: 3 nodes with 2 CPUs & 2 GB memory each
• Bats (Bash Automated Testing System) plus assertion libraries for kubectl
• GitHub Action that handles scheduling/queuing of subsequent jobs against shared cluster

Anatomy of a Test Case

The idea is as simple as compelling. Runme’s CLI allows referencing code snippets by name (explicitly defined in command block annotations for clarity). This mirrors precisely what’s being presented to developers on the website’s Getting Started section to copy & paste into their terminals. It goes without saying that Runme’s UX is much more elegant than copy & paste.

Asserting Test Conditions

Inside our Bats files, we replicate the sequence of steps and the contained instructions as test cases. We can define pre-conditions, execute the Markdown command block, and check post-conditions for returned exit code, output, and desired Kubernetes cluster state. The Bats DETIK assertion library does a phenomenal job of expressing Kubernetes assertions fluently. Bats is also capable of handling the setup and teardown of a test suite. Feel free to ask us about the details.

@test "Verify linkerd injection (step 4)" {
  DETIK_CLIENT_NAMESPACE="emojivoto"
  run "runme run kubectl-get"
  assert_line -p "deployment \"emoji\" injected"
  assert_line -p "deployment \"vote-bot\" injected"
  assert_line -p "deployment \"voting\" injected"
  assert_line -p "deployment \"web\" injected"
  try "at most 10 times every 30s to get pods named 'emoji' and verify that 'status' is 'running'"
  try "at most 10 times every 30s to get pods named 'vote-bot' and verify that 'status' is 'running'"
  try "at most 10 times every 30s to get pods named 'voting' and verify that 'status' is 'running'"
  try "at most 10 times every 30s to get pods named 'web' and verify that 'status' is 'running'"
  assert_success
}

While Bats will support parallel execution, we want these test cases to run serially in a predetermined order. Humans run one command at a time. And, in stateful nature, downstream execution results depend on completing prior commands. While your run-unit-tests-in-parallel-self might object at first, this is, in fact, a good thing. It replicates how developers consume software docs in reality.

The same idea is transferrable to e.g., internal repos that house libraries, apps, or services. Imagine every repo’s README.md, BUILD.md, or DEPLOY.md being tested this way.

Let Us Know What You Think

While we’re happy with our demo’s results, there is much room for improvement. Here are some items we have discussed and would love your feedback on:

1. Leverage notebook-serialized cell output for test assertions? The idea is to record cell outputs stored alongside the inputs and provide the functionality to define common assertions, e.g., substrings, expected exit code, or fuzzy threshold diffing.

2. Runme-integrated test-harness? While Bats fits the demo’s use case well, it is likely desirable to use Runme’s notebook parser and structural awareness to maintain a test suite inside of notebooks instead of describing them separately, in this case, *.bats files.

3. Provide an official Runme Github Action for easy integration? Significantly reduce boilerplate, ideally where the workflow auto-detects Runme-enabled Markdown files inside of the repo and runs the associated test suite.

The common theme in the ideas above is to absorb commonly required capabilities to express test cases into Runme and minimize overhead. Unlock docs testing value faster. The better we understand your use cases, the more effectively we can evolve Runme following this philosophy. Please, we urge you, get in touch!

Get Involved

Outside of CI/CD-related features, the team is currently heads-down working through Runme's final touches of the Notebook Terminal output UX (experimentally available for non-interactive and background tasks) directly inside your READMEs. Beyond that, our roadmap is full of UX improvements, big and small, to continue streamlining the daily workflows of developers.

Now go try it out and help us prioritize! We’d love to hear what you think. If you run into any problems, please don’t hesitate to report them as a GitHub issue or talk to us on Discord.

Happy Docs Testing 🤩!