What You See is What You Get - Building a Verifiable Enclave Image

Table of Contents

 1. Obstacle of proofing TEE
       1.1. Image digest is meaningless
       1.2. Stable image digest is difficult
 2. Solution - Trusted build pipeline
       2.3. GitHub provides the service suite we need
       2.4. Use SigStore to sign and endorse the image
       2.5. Putting everything together
       2.6. How can service consumers verify the PCRs
 3. What's beyond
       3.7. Build log retention
       3.8. Build pipeline still needs to be simple
 4. Wrap up

---

Link to the GitHub Action discussed in this post: https://github.com/marketplace/actions/aws-nitro-enclaves-eif-build-action

---

AWS Nitro Enclaves is a Trusted Execution Environment (TEE) where service consumers can validate if the environment is running what it claims to be running.

I've posted previously on how to achieve it by using attestation documents and why should we care about the content of the attestation document:

• How to Use AWS Nitro Enclaves Attestation Document
• AWS Nitro Enclaves Ecosystem (1) - Chain of trust

In this blog post, I want to dive deep into achieving zero-trust between service providers and consumers on TEE, particularly AWS Nitro Enclaves.

Obstacle of proofing TEE

Image digest is meaningless

Platform configuration registers (PCRs) are just the application image digests; they are generated by a one-way hashing function against the image.

We cannot see what is inside the image by looking at the hash value. So without knowing what generated the PCRs, it's meaningless.

For service consumers who have no oversight of the application source code and build process, they have nothing to do, even if they can validate the attestation document. They can only trust whoever saying "This PCR value 'abcdef' is generated by a secure and safe application"

Service providers may ask 3rd party auditor to attest the above statement. But it's no different than getting SOC2 or ISO 27001 certified.

If we are satisfied with this level of trust model, we can stop talking about TEE already. Why don't we send the SOC2 certificate to the consumers instead of the attestation document?

Stable image digest is difficult

If service consumers can access the application source code and the build pipeline definition, they may build the enclave image and compare the digest with the one provided in the attestation document.

The problem is that generating a stable image digest is difficult, even a small trivial difference occurs in build time can make the digest entirely different.

Some common trivial changes in build time are:

1. Timestamp

   Some build steps inject the current timestamp into the environment (e.g. embedded timestamp in .pyc files when installing Python dependencies).

   This makes the resulting image dependent on the time of build.

2. External dependencies

   Even if we pin all dependencies to the exact version, using external sources may still cause image differences.

   E.g., when running apt update on Ubuntu, the manifest pulled from an external source may be different than previously pulled.

3. Other build time randomness

   There are more examples that can cause image differences.

   E.g., Using random strings as temporary folder names.

By looking at the image digest difference, we cannot tell if it's caused by trivial differences or service provider changing their source code.

Solution - Trusted build pipeline

To avoid the hiccup of creating a reproducible build process, we can instead create a trust build pipeline that service consumers can see and trust.

To make it work on AWS Nitro Enclaves images, I have created a GitHub action AWS Nitro Enclaves EIF Build Action

GitHub provides the service suite we need

To achieve an end-to-end chain of trust from source code, build process, to the resulting enclave image, we need a publicly accessible and trusted code repository, build environment, and artifact store.

Undoubtedly, GitHub is currently the most popular platform to host open-source code. GitHub also provides GitHub Actions as the build environment and GitHub Packages as the artifact store.

By putting the entire build pipeline into GitHub, we can minimize the number of parties we build trust into.

Use SigStore to sign and endorse the image

The other main component of the solution is SigStore.

SigStore is a set of open-source technologies to handle the digital signing of software.

Using SigStore, we can easily sign the enclave image and prove to the public that this image is built by a specific pipeline run, from a particular code repository commit.

Putting everything together

In this sample repository, I use the AWS Nitro Enclaves EIF Build Action to build a Nitro Enclave image from the source code.

After the artifacts are built and pushed to the GitHub Container Registry (GHCR), there will be a cosign command to sign the artifact.

Several things are happening behind this command:

1. The OIDC token of the GitHub workflow run is used to request a signing certificate from Fulcio

2. The digest of the uploaded artifacts (In this scenario, the Nitro Enclave EIF and its information) is signed

3. The signature is pushed to the artifact store (i.e., GHCR)

4. The signing certificate and the artifact signature are recorded in the Rekor transparency log

How can service consumers verify the PCRs

Service consumers can audit the code once the artifact is signed and pushed to the registry.

To verify the PCRs they get from the attestation document are indeed the same as what was built by the said build pipeline, they can do the following:

1. Use cosign to verify the artifact against the signature stored in Rekor
   

   cosign verify ghcr.io/username/repo:tag \
       --certificate-identity-regexp https://github.com/<username>/<repo>/ \
       --certificate-oidc-issuer https://token.actions.githubusercontent.com
   

   

2. Validate the information on the signing certificate

   User can search the signing entry on Rekor Search by its log index

   

   

   We should look carefully at the following attributes:

   1. OIDC Issuer: The token must be issued by the trusted build environment.

      (In this example, it must be the GitHub Actions OIDC issuer https://token.actions.githubusercontent.com)

   2. GitHub Workflow SHA: This indicates which particular Git commit the build pipeline run is from.

      This helps us identify from which commit we should look at when auditing the source code.

   3. Build Config URI: This file defines the build workflow.

      We should also check if the build configuration is safe, just like how we audit the application code.

   4. Runner Environment: We should also ensure the build was run on GitHub-hosted runners instead of self-hosted ones that cannot be trusted.

3. Audit the code based on the information on the certificate

   After knowing how the artifact was built, we can go to the specific commit of the code repository to audit the codes.

4. Pull the artifact and get the PCRs

   After all the validation, we can use ORAS to pull the EIF and its information.

   The PCR values are inside the signed text file; they can be compared with the ones given by the attestation document from the running service.
   

   oras pull ghcr.io/username/repo:tag@sha256:<digest>
   

   

What's beyond

Build log retention

GitHub actions run on public repositories can be viewed by anyone; it gives service consumers more confidence in the enclave application by looking into how exactly it was built.

However, the GitHub action log can only be retained for up to 90 days.

If the service consumers want utmost scrutiny over the enclave application, service providers may need to rebuild the enclave image every 90 days so that build logs can be audited at any point in time.

Build pipeline still needs to be simple

Although service consumers can audit the build process in this design, it doesn't mean service providers don't need to make their build process simple.

The more complex a build pipeline is, the more difficult it can be to understand what's being done under the hood.

E.g., If the build pipeline pulled source codes from an external source instead of the source code repository; How can we see, from the build log, what the content of those codes is?

Wrap up

Three years after AWS announced Nitro Enclaves, the support from AWS is still minimal. (Sidetrack: My PR on kmstool is still pending for review)

There is still little to no discussion on how to utilize Nitro Enclaves to achieve TEE in the real world. I hope the tools I build can at least offer some help to the community.

Link to the GitHub Action: https://github.com/marketplace/actions/aws-nitro-enclaves-eif-build-action