So you’ve set up continuous integration for your project. Everything looks good and now all you need is a container. Just build and run it, right? Not so fast!

Whether using containers to support development or for packaging an application, it’s easy to take them for granted. But many things can go wrong with them: moved files, incorrect permissions, a user is missing, the Dockerfile is incomplete, the list goes on…

Containers are as crucial as the code they support. In this tutorial, we’ll introduce a different way of testing them before deployment.

One of the many ways Google tests containers

Container Structure Tests (CST) is a container-testing tool developed by Google and open-sourced with the Apache 2.0 license. CST comes with a predefined set of tests for looking at what’s actually inside a container image.

For instance, CST can check if a file exists, run a command and validate its output, or check if the container exposes the correct ports. Almost every declarative keyword in the Dockerfile has a corresponding test.

One important note is that the project is not officially supported by Google, so it doesn’t show a lot of activity. But it’s popular enough to keep it active and it’s still accepting contributions.

More tests, less uncertainty

Any old container should do in order to try out CST. Though it will be easier if we build one from a known Dockerfile. So, if you want to follow along with me as I explore this tool, fork and clone our Ruby Kubernetes demo project:

bundle install --path vendor/bundle
bundle exec rackup

bundle exec rspec

docker build -t semaphore-demo-ruby-kubernetes
docker run -p 80:4567 semaphore-demo-ruby-kubernetes
curl localhost
> hello world :))

It’s a “Hello, World” application written on Ruby. It comes with a Dockerfile and a complete CI/CD pipeline.

You’ll also need a:

• A Docker installation.
• A CST config file.
• The CST executable.

Once you’ve cloned the demo, build the container image with:

$ docker build -t test-image .

Install the CST tool using the installation instructions. For instance, on Linux:

$ curl -LO https://storage.googleapis.com/container-structure-test/latest/container-structure-test-linux-amd64 && chmod +x container-structure-test-linux-amd64 && sudo mv container-structure-test-linux-amd64 /usr/local/bin/container-structure-test

If you’re running on a non-Intel architecture, you can build the project by yourself or try the CST image instead.

The command to run the test follows works like this:

$ container-structure-test test –config config-cst.yaml –image test-image:latest

But of course, that won’t work until we configure the tests. We’ll do that next.

Setting up CST tests

CST supports three categories of tests:

• Commands: starts the container, runs the command, and validates its result.
• Filesystem: checks for file existence, owner, permissions, and contents.
• Metadata: this category contains things such as environment variables, exposed ports, labels, among other image metadata.

Create a new file called config-cst.yaml (JSON also works) and add the following mandatory line:

schemaVersion: 2.0.0

We’ll start with the command tests.

Command tests

Let’s try some command tests. We can use something like this to check that Ruby is installed. Of course, I cheated and looked where it’s actually located before trying.

commandTests:
  - name: "Ruby is installed"
    command: "which"
    args: ["ruby"]
    expectedOutput: ["/usr/local/bin/ruby"]

Now we’ll check that ruby --version outputs the correct number:

  - name: "Ruby version is correct"
    command: "/usr/local/bin/ruby"
    args: ["--version"]
    expectedOutput: ["ruby 2.7.*"]

Should we be in a really security-conscious mindset, we can checksum the Ruby binary for extra safety. We can run preparation commands before the test with setup.

  - name: "Ruby binary checksum"
    setup: [["apt-get", "update"], ["apt-get","install","-y","shatag"]]
    command: "sha512sum"
    args: ["/usr/local/bin/ruby"]
    expectedOutput: ["df2fb393261ab88e5f991b96d958363f5e5185b51f1af319375be0d8b9ed6c27097ac8bfab399798497909c3e9e2bcc6d715a1e514f13fe5b7365344af555c7e  /usr/local/bin/rub"]

Now that we have some initial tests, we can actually run the tool for the first time:

$ container-structure-test test --config config.yaml --image test-image:latest

=== RUN: Command Test: Ruby is installed
--- PASS
duration: 391.76725ms
stdout: /usr/local/bin/ruby

=== RUN: Command Test: Ruby version is correct
--- PASS
duration: 319.6335ms
stdout: ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [aarch64-linux]

=== RUN: Command Test: Ruby binary checksum
--- PASS
duration: 302.199834ms
stdout: df2fb393261ab88e5f991b96d958363f5e5185b51f1af319375be0d8b9ed6c27097ac8bfab399798497909c3e9e2bcc6d715a1e514f13fe5b7365344af555c7e  /usr/local/bin/ruby


===============================
=========== RESULTS ===========
===============================
Passes:      3
Failures:    0
Duration:    1.013600584s
Total tests: 3

All command tests require a working Docker installation, as the tool must start a temporary container to run them. However, filesystem and metadata tests can be run without Docker appending the --driver tar option.

Filesystem tests

Filesystem tests inspect the contents of the image; they check if files exist, their permissions, their contents, owner, and group.

Here’s how we can test that the code was correctly copied in the image. If we look at our Dockerfile, it should live in the /app/ folder. So, we define a fileExistenceTests like this:

fileExistenceTests:
  - name: 'app.rb exists and has correct permissions'
    path: '/app/app.rb'
    shouldExist: true
    permissions: '-rw-rw-r--'
    uid: 0
    gid: 0

We can also check the reverse: that a file is not present in the image. Setting shouldExist to false is a great way to avoid shipping sensitive files by mistake. For example, we don’t need the unit tests contained in spec for the final build.

  - name: 'spec/ directory should not exist'
    path: '/app/spec'
    shouldExist: false

This time, CST should fail because the Docker image has the spec folder.

$ container-structure-test test --config config.yaml --image test-image:latest

=== RUN: Command Test: Ruby is installed
--- PASS
duration: 308.371875ms
stdout: /usr/local/bin/ruby

=== RUN: Command Test: Ruby version is correct
--- PASS
duration: 312.744792ms
stdout: ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [aarch64-linux]

=== RUN: Command Test: Ruby binary checksum
--- PASS
duration: 286.408167ms
stdout: df2fb393261ab88e5f991b96d958363f5e5185b51f1af319375be0d8b9ed6c27097ac8bfab399798497909c3e9e2bcc6d715a1e514f13fe5b7365344af555c7e  /usr/local/bin/ruby

=== RUN: File Existence Test: app.rb exists and has correct permissions
--- PASS
duration: 0s
=== RUN: File Existence Test: spec/ directory should not exist
--- FAIL
duration: 0s
Error: File /app/spec should not exist but does

===============================
=========== RESULTS ===========
===============================
Passes:      4
Failures:    1
Duration:    907.524834ms
Total tests: 5

We can fix the spec failure by adding the following line into the .dockerignore (you may also experience a permission error in /app/app.rb, which can be quickly fixed with chmod).

spec/

After rebuilding the image, the test should pass.

We have tested that a file exists. But, what about its contents? For that, we should use fileContentTests. The following example shows how to test if the Ruby Gems have been installed from a safe repository.

fileContentTests:
  - name: 'Gemfile remote is rubygems.org'
    path: '/app/Gemfile.lock'
    expectedContents: ['remote: https://rubygems.org/']

Both types of tests support regular expressions in the expected* fields for more flexibility.

Metadata tests

Unlike the others, you can only have one metadata test. But it may check several things simultaneously, as metadata tests include a whole range of standard Docker variables.

Let’s say we want to test environment variables. In that case, we use env.

metadataTest:
  env:
    - key: APP_HOME
      value: /app
    - key: RUBY_VERSION
      value: 2.7.4

You can also check Docker declarations like WORKDIR, EXPOSE, VOLUME, or USER.

  exposedPorts: ["4567"]
  workdir: "/app"
  volumes: []

And the always important CMD and ENTRYPOINT.

  cmd: ["bundle", "exec", "rackup", "--host", "0.0.0.0", "-p", "4567"]
  entrypoint: []

Once satisfied with the tests, commit the config file into the repository. This is the final version I got after experimenting with some extra tests follows.

schemaVersion: 2.0.0
commandTests:
  - name: "Ruby is installed"
    command: "which"
    args: ["ruby"]
    expectedOutput: ["/usr/local/bin/ruby"]
  - name: "Ruby version is correct"
    command: "/usr/local/bin/ruby"
    args: ["--version"]
    expectedOutput: ["ruby 2.7.*"]
  - name: "Ruby binary checksum"
    setup: [["apt-get", "update"], ["apt-get","install","-y","shatag"]]
    command: "sha512sum"
    args: ["/usr/local/bin/ruby"]
    expectedOutput: ["df2fb393261ab88e5f991b96d958363f5e5185b51f1af319375be0d8b9ed6c27097ac8bfab399798497909c3e9e2bcc6d715a1e514f13fe5b7365344af555c7e  /usr/local/bin/rub"]
  - name: "Bundle is installed"
    command: "which"
    args: ["bundle"]
    expectedOutput: ["/usr/local/bin/bundle"]
  - name: "Bundler version is correct"
    command: "/usr/local/bin/bundle"
    args: ["--version"]
    expectedOutput: ["Bundler version 2.1.*"]
fileExistenceTests:
  - name: 'app.rb exists and has correct permissions'
    path: '/app/app.rb'
    shouldExist: true
    permissions: '-rw-rw-r--'
    uid: 0
    gid: 0
  - name: 'spec/ directory should not exist'
    path: '/app/spec'
    shouldExist: false
fileContentTests:
  - name: 'Gemfile remote is rubygems.org'
    path: '/app/Gemfile.lock'
    expectedContents: ['remote: https://rubygems.org/']
metadataTest:
  env:
    - key: APP_HOME
      value: /app
    - key: RUBY_VERSION
      value: 2.7.4
  exposedPorts: ["4567"]
  cmd: ["bundle", "exec", "rackup", "--host", "0.0.0.0", "-p", "4567"]
  workdir: "/app"
  entrypoint: []
  volumes: []

Testing container structure in CI/CD

Of course, CST wouldn’t be a lot of help to us unless it’s part of a CI/CD pipeline. The logical place for structure tests is between container build and deployment.

Adding CST in your CI/CD pipeline is straightforward. To keep things simple, we’ll extend the one already included in the demo, but the steps should work with any pipeline.

First, ensure Semaphore has access to your repository. Follow the getting started guide to learn how to do this.

The demo pipeline builds and tests the Ruby app, then dockerizes it and finally deploys it to Kubernetes. We’ll add a structure test right between Docker build and Kubernetes deploy.

First, ensure you have stored your Docker Hub credentials as a Semaphore secret.

Next, open the workflow editor using the Edit Workflow button.

Expand the continuous delivery pipeline and add a block immediately after the Docker build step.

The CST block will have one job with five commands:

1. Sign in with Docker Hub, where the container image is stored.
2. Pull the image into the CI environment.
3. Install the CST Linux binary.
4. Clone the repository so we can access the CST config file.
5. Run the tests. If the test fails, the process stops with an error.

The complete command sequence is:

echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
docker pull "${DOCKER_USERNAME}"/semaphore-demo-ruby-kubernetes:$SEMAPHORE_WORKFLOW_ID
curl -LO https://storage.googleapis.com/container-structure-test/latest/container-structure-test-linux-amd64 && chmod +x container-structure-test-linux-amd64 && sudo mv container-structure-test-linux-amd64 /usr/local/bin/container-structure-test
checkout
container-structure-test test --config config-cst.yaml --image "${DOCKER_USERNAME}"/semaphore-demo-ruby-kubernetes:$SEMAPHORE_WORKFLOW_ID

To finalize, enable the dockerhub secret on the block and give it a try by clicking on Run the workflow.

Once the CI pipeline is complete, an auto-promotion should kick off the continuous delivery pipeline.

The image is ready and tested for the next stage. Now you can deliver containers with more confidence.

Conclusion

The more you know about your container, the less surprises you’ll get. Container Structure Test may not be the most flexible tool, but it certainly is a quick and easy way of adding some confidence to the release process. So, it should be on the radar of anyone using containers for serious work.

Read next:

• Kubernetes vs Docker: Understanding Containers in 2021

• Beyond Docker with Earthly

• Docker Image Size – Does It Matter?