Securing Containers with Seccomp: Part 2

This blog post is the second half of my research on making it easier to create Seccomp-BPF filters and secure your containers with them. If you haven’t read the first half of this blog post yet, I recommend doing so before continuing here.

For returning readers, I’ll briefly refresh you on my last post. I introduced Seccomp-BPF, which is a Linux kernel feature that allows you to create filters that restrict which syscalls a process can make. This allows you to prevent a process from being able to do something that it was not intended to do if it requires a new syscall. Seccomp-BPF filters can also be applied to container runtimes and orchestration tools to limit what your containers can do while they’re running. That gives you an opportunity to add an additional layer of protection against the potential exploitation of your applications and services.

The biggest roadblock with learning about and popularizing the use of this awesome feature has been the lack of tools and processes that bridge a low-level kernel feature like Seccomp-BPF to modern software development practices. In my last article, I explored a tool created by Red Hat called oci-seccomp-bpf-hook. It helps simplify the creation of Seccomp filters by recording syscalls made by a container during runtime and pumping them into a filter that you can use with Seccomp. The OCI hook dramatically reduces the knowledge about syscalls that you’d need to create a Seccomp filter, but there’s still quite a bit of overhead involved in updating your Seccomp filter with the tool every time you update your code.

To solve that problem, I’m going to show you how to automate the creation of a Seccomp filter for your app every time your code gets updated. We’re going to create a Continuous Integration (CI) workflow via GitHub Actions that utilizes Red Hat’s oci-seccomp-bpf-hook. Let’s dig in!

The Example Application

To paint a picture of what this automation looks like in the context of modern software development, I created an example application. The application is a simple Python flask API service. You can find all the example and workflow code in my GitHub repo, and you can fork the project if you want to test it yourself.

The REST API has two endpoints. One is at /dummy and is just a dummy function that responds successfully if you send it “valid input :)”.

@app.post("/dummy")
def dummy():
    """ Dummy function for testing """
    if missingParams := missing(request, data=["input"]):
        return {"error": missingParams}, 400
    if request.json["input"] == "valid input :)":
        return {"success": "nice input"}, 200
    else:
        return {"error": "bad input"}, 400

The API’s second endpoint was made to be intentionally vulnerable for the purpose of this demonstration. The vulnerable API call is at /command and will run any command sent to it in a shell subprocess.

@app.post("/command")
def command():
    """ Execute the command and return the output """
    if missingParams := missing(request, data=["command"]):
        return {"error": missingParams}, 400
    command = request.json["command"]
    process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = process.communicate()
    return {"stdout": stdout.decode("utf-8"), "stderr": stderr.decode("utf-8")}

A Dockerfile was also created for the application so it can be run in a container and used with oci-seccomp-bpf-hook. Now that we have a containerized application to generate a Seccomp filter for, let’s work on automating the creation of the filter.

Setting up the Actions Runner

The first thing we need to do is set up our self-hosted Actions Runner on a Fedora server. If you remember from the first blog post, oci-seccomp-bpf-hook was made to run with Podman, Red Hat’s container runtime. GitHub’s default automation runners are Ubuntu-based, and even though we could use them and install podman and the OCI hook from source, it’s a lot easier, in the long run, to use our own Fedora server with the tools pre-installed. This shouldn’t be a problem to set up in a professional software development environment.

If you want to quickly stand up a build server to test this tutorial, you can use a virtual machine, an old desktop/laptop, or even a Raspberry Pi to host your Fedora server. You just need to make sure it’s connected to the internet when your automation kicks off.

Once you have a Fedora server, you’ll need to create a secure account on it for your GitHub Actions to use when running. First, let’s become the root user to simplify the rest of the setup:

sudo su

Next, we’ll create an unprivileged account with a home directory:

useradd -m github

And create a secure password for the account:

passwd github

Now we need to make sure that the ‘github’ user has the proper permissions to run the programs it needs. To do that, we’ll use ‘visudo’ to edit the /etc/sudoers file:

visudo

Add the following line to the bottom of the sudoers file to allow the user ‘github’ to execute dnf and podman as root without needing a password:

github ALL=(ALL) NOPASSWD: /usr/bin/podman, /usr/bin/dnf

The reason we must not require a password is because GitHub actions can’t enter a password while automating our steps. Once we have the sudoers entry saved, we are done with our Fedora setup for now.

Next, it’s time to set up the Actions runner. For this part, you’ll need to make sure that your repository is set to Private rather than Public. For security reasons, GitHub doesn’t allow you to use self-hosted runners on public repositories unless you are an organization. To begin setting up our self-hosted runner, we’ll need to go to our GitHub repository and click “Settings.”

In the repository settings, go to Actions > Runners. From there, click the green “New self-hosted runner” button in the top right.

Select Linux as the operating system, and select the CPU architecture that your Fedora server is running on. If you are using a Raspberry Pi, you’ll need to do ARM64. Otherwise, you are probably using x64. After that, we need to follow GitHub’s instructions on our Fedora server as the ‘github’ user that we just set up. To switch accounts, you just need to run the following command:

su github

When configuring the GitHub runner, press Enter to accept both the default runner group and the default runner name. When it asks if you want to add additional labels, type ‘fedora’ and then hit Enter. Lastly, press Enter to accept the default work folder.

After configuring the runner, it should be registered with our GitHub repo. We need to make sure to finish the instructions from GitHub and launch the runner with the provided script. The ‘run.sh’ script will make the runner available for Actions to be deployed to it. Once the runner is all set up, we’re ready to set up our automation!

Creating the Seccomp Filter Generation Workflow

GitHub Actions are kept in a specific folder of your repository at {root}/.github/workflows. You need to define each workflow in the YAML file format. In our example repo, I created a file called “seccomp.yml” and put it in the Actions folder. This is how the file starts:

name: "Generate Seccomp Filter"

on:
  push:
    paths-ignore:
      - seccomp.json

At the top of the file, we name our workflow “Generate Seccomp Filter” and specify when our automation should kick-off. We are going to run the workflow every time we push code to the GitHub repo, except when the only change is the “seccomp.json” file. The exception prevents GitHub Actions from running the workflow again after generating the Seccomp filter.

Next, we start to define the actions that need to happen in our workflow:

jobs:
  generate-seccomp-filter:
    runs-on: fedora  # self-hosted

“Generate-seccomp-filter” is the name of the action we are about to create, and we specify that the action runs on our self-hosted Fedora server that we set up earlier. The last thing to do is to define the steps we want to take during our automation. First, we want to check out our code, make sure we have the tools we need installed and build the container image that holds our flask API:

steps:
  - name: Check out repository
    uses: actions/checkout@v2
  - name: Install tools
    run: sudo dnf install podman oci-seccomp-bpf-hook
  - name: Build Container
    run: sudo podman build --tag demo -f ./Dockerfile

Next, we need to run the container and record the syscalls it makes with the OCI hook… But wait, we’re missing something… How do we make sure that our application makes all the syscalls that it needs while we are recording? If we just launch our API server, that doesn’t necessarily record the syscalls that are made when an API endpoint is called. Fortunately, there’s a solution to this problem. We’ll run our unit and functional tests while recording syscalls with the OCI hook!

A mature software development lifecycle includes unit tests written for the application to ensure code quality. We can piggyback off that effort and use the automated tests to make sure we are hitting all the functionality of our application while we record syscalls. For our example app, I wrote unit tests using pytest and have a script that kicks them off. In our Action workflow, we can run the tests in our container, capture the syscalls in a Seccomp filter, and commit the filter back to our GitHub repository:

- name: Record syscalls
  run: "sudo podman run --annotation io.containers.trace-syscall=of:/tmp/seccomp.json demo /opt/demo/test.sh"

# Commits updated seccomp filter back to the same branch (commit || echo to not fail on no changes)
- name: Commit seccomp filter to repo
  run: |
    cp /tmp/seccomp.json .
    git config --global user.name "github-runner[bot]"
    git config --global user.email "github-    runner[bot]@users.noreply.github.com"
    git add .
    git commit -m "generated new seccomp filter" || echo
    git push origin ${GITHUB_REF##*/} -f

That’s the end of our seccomp.yml workflow file. Now every time we push new code to our GitHub repository, it will generate a Seccomp filter for our app! It required some setup to get to this point, but now that it’s done there’s no more effort required to keep an up-to-date Seccomp profile for our application. Below you can see a screenshot from my example repo of the workflow running successfully after I updated my repository’s README.

Conclusion

You now have everything you need to automatically create your own application-specific syscall whitelist filters. Running your applications with the generated Seccomp-BPF filters will dramatically reduce the risk of them being successfully exploited. The best part of this demonstration is that the GitHub Actions workflow that we created isn’t specific to my example app – it’s mostly generic and can be easily modified for any other project. You only need to make sure that your ‘podman build’ and ‘podman run’ steps reflect the way you build your container image and run your tests. Other than that, you just need to keep in mind one requirement.

The important thing to remember when generating Seccomp filters for your application is that you NEED to have a very high percentage of your code getting executed in your automated unit and functional tests. If your application has some functionality that isn’t being executed when you are generating the Seccomp filter, there’s a chance that you could be missing some syscalls in your filter. This means they will get blocked when your code tries to run with the Seccomp filter applied.

Luckily, there’s a tip-off that will indicate that your application isn’t working because of Seccomp. If Seccomp is preventing your app from making a needed syscall, the error that your application throws will always include a message about not having permission to do something. In that case, you just need to figure out which part of your code you missed in your automated testing. Then you add a test for the missed case, and your new Seccomp filter will be generated as soon as you push the new test to your repo. There are tools out there that help you monitor your test coverage, and I recommend you use them if you are generating Seccomp filters for your app.

That concludes the demonstration on how to automate the generation of Seccomp filters. Because there aren’t many resources out there for Seccomp yet, this is a new concept for most developers and security professionals. However, the security gained from this practice could be a game-changer for businesses trying to protect their mission-critical services. The technology already exists to make Seccomp a standard. I hope that my explanations and examples have made it easier to dive into this new way of securing your applications!