loading...

Kubernetes Integration Testing

danquack profile image Daniel Quackenbush ・4 min read

Summary

Infrastructure pipeline testing is essential for ensuring minimal regression, healthy systems, and faster mean time to recovery in patches. I have found it especially useful to perform integration tests to validate environments pre-deployment, to ensure they meet all the technical and compliance requirements.

Use Case

For the test, I have created a sample flask app, deployed on EKS, that will reach out to S3 and grab a file called "test". The flask-app manifest defines a readiness probe to ensure the /test endpoint (that grabs the file from s3) can fire successfully. Our inspec test suite will then ensure the pod deploys and starts. The pytest suite will validate k8s components like IAM authorization through our readiness probe, networking, and has the potential to expand to end to end tests in the future.

Alt Text

If deploying this code set, ensure to replace any <VAR> with an appropriate value for your environment.




Technologies

Inspec

Despite open source offerings, such as inspec-k8s, the backing sdk was written by a company who closed its doors in 2019. Depending on the backing runner for your cluster creation, you can also run into dependencies mismatch, creating a nonstarter execution, see dry types PR. Ultimately, it is possible, but several workarounds are needed in getting started.

To fully utilize inspec, you will have to deploy your application configuration first, as well as install the Kubernetes train. Alternatively, running from bgeesaman's inspec-k8s-runner docker container gives you a nice starting place. Once installed, you can begin to test the various components within your namespace.

Below are some sample resources you can utilize to test against.

### Test your pods exists, and all are running
control "k8s-app-validate" do
  impact 1.0
  title "Validate K8s test Application"
  desc "The k8s-app test app should exist and be running"

  ### Test various namespaces exist
  describe k8sobject(api: 'v1', type: 'namespaces', name: 'default') do
      it { should exist }
  end
  ### Test for every pod in deployment the pod exists, and is running
  k8sobjects(api: 'v1', type: 'pods', namespace: 'default', labelSelector: 'app=flask').items.each do |pod|
    describe "#{pod.namespace}/#{pod.name} pod" do
      subject { k8sobject(api: 'v1', type: 'pods', namespace: pod.namespace, name: pod.name) }
      it { should exist }
      it { should be_running }
    end
  end
  ### Test your service exists
  describe k8sobjects(api: 'v1', type: 'services', namespace: 'default', labelSelector: 'app=flask') do
    it { should exist }
  end
end

Pytest

Vapor has a framework called kubetest, which will deploy a configuration to your cluster in a unique namespace. This framework is meant for deploying and testing, and not necessarily testing existing infrastructure that exists. One of the advantages of this framework is it natively has readiness checks, whereas inspec did not.

Once deployed, you can utilize standard pytest execution to assert against those resources. Below walks through a few scenarios, which will use the configurations provided above.

There are some limitations in the API objects supported, see API Resources.

from boto3 import client
from time import time

# Create And Test Service Account
def create_sa(kube, modifier):
    """
    A helper function to create service account
    """
    sa = kube.load_serviceaccount("configs/sa.yaml")
    account_id = client('sts').get_caller_identity()["Account"]
    role_arn = f"arn:aws:iam::{account_id}:role/test_role"
    sa.obj.metadata.annotations['eks.amazonaws.com/role-arn'] = role_arn
    return sa

def test_create_sa(kube, modifier):
    """
    A function to test the creation of a service account
    Goal: This will test the ability to interface with the k8s cli
    """
    sa = create_sa(kube, modifier)
    kube.create(sa)
    assert sa.is_ready()
# Create and Test Deployment
def create_deployment(kube, sa, modifier):
    """
    A helper function to create a deployment object
    """
    account_id = client('sts').get_caller_identity()["Account"]

    deployment = kube.load_deployment('configs/deployment.yaml')
    repository = f"{account_id}.dkr.ecr.us-east-1.amazonaws.com/k8s-test"
    deployment.obj.spec.template.spec.containers[0].image = repository
    deployment.obj.spec.template.spec.service_account_name = sa.obj.metadata.name
    deployment.obj.spec.template.spec.containers[0].env[1].value = f"quackenbush-test-bucket"
    return deployment

def test_deployment(kube, modifier):
    """
    A function to test the creation of a deployment.
    Goals: If a pod becomes ready, that means it can successful connect to SSM
    """
    sa = create_sa(kube, modifier)
    kube.create(sa)

    deployment = create_deployment(kube, sa, modifier)
    kube.create(deployment)

    timeout = time() + 60 # 60 seconds
    while not deployment.is_ready():
        if time() > timeout:
            # Fail early
            assert deployment.is_ready()

    pods = deployment.get_pods()

    for pod in pods:
        pod.wait_until_containers_start(timeout=60)
        timeout = time() + 300 # 5 Minutes
        while not pod.is_ready():
            if time() > timeout:
                break

        assert pod.is_ready()

Go

Intel wrote a blog post, describing a go framework, Ginkgo, which is worth diving into, but it was excluded due to the adoption path it would take onboarding the team to a new language.

Discussion

markdown guide