In this blog, I will talk about the generation of custom/new chaos experiment with the help of Litmus SDK . Before jumping in, let's do a quick recap on Litmus. Litmus is a framework for practicing Chaos Engineering in cloud-native environments. Litmus provides a chaos-operator, a large set of chaos experiments in its HUB, detailed documentaion, quick demo, and a friendly community.

What is Litmus SDK?

The Litmus SDK provides a simple way to bootstrap your experiment and helps create the aforementioned artifacts in the appropriate directory (i.e., as per the chaos-category) based on an attributes file provided as input by the chart-developer. The scaffolded files consist of placeholders which can then be filled as desired.

It generates the custom chaos experiments with some default Pre & Post Chaos Checks (AUT & Auxiliary Applications status checks). It can use the existing chaoslib (present inside /chaoslib directory), if available else It will create a new chaoslib inside the corresponding directory.

Life Cycle of a Chaos Experiment

Each Chaos Experiment is divided into six main sections:

• Prepare: Prepare section contains the initialization steps (get ENVs), Updation of chaos result to marked the beginning of the chaos experiment.
• PreChaosCheck: PreChaosCheck contains some default checks (AUT & Auxiliary Application status check) and custom checks, which may vary on the basis of experiment i.e, liveness check, data persistence check, etc.
• ChaosInject: ChaosInject invoke the actual chaos, It contain the main business logic.
• CleanUp: CleanUp contains the steps to remove the helper/external pod, if any
• PostChaosCheck: PostChaosCheck contains the similar steps as the PreCheckCheck section have. It used to ensure the resiliency after chaos injection.
• Summary Summary updates the verdict(Pass/Fail) inside the chaos result and FailStep if the experiment fails.

Pre-requisites

Go Experiments

• go should be is available & the GOPATH env is configured appropriately

Ansible Experiments

• python3 is available (sudo apt-get install python3.6)
• jinja2 & pyYaml python packages are available (sudo apt-get install python3-pip, pip install jinja2, pip install pyYaml)

Steps to Generate Experiment via Litmus SDK

• Clone the litmus-go repository for go experiments and litmus-ansible for the ansible experiments and navigate to the contribute/developer-guide folder.

## for litmus-go
$ git clone https://github.com/litmuschaos/litmus-go.git
$ cd litmus-go/contribute/developer-guide

## for litmus-ansible
$ git clone https://github.com/litmuschaos/litmus-ansible.git
$ cd litmus-ansible/contribute/developer_guide

• Populate the attributes.yaml with details of the chaos experiment (or chart). Use the attributes.yaml.sample as reference.

As an example, let us consider an experiment to kill one of the replicas of an Nginx deployment. The attributes.yaml can be constructed like this:

$ cat attributes.yaml 

---
name: "sample-pod-delete"
version: "0.1.0"
category: "sample-category"
repository: "https://github.com/litmuschaos/litmus-go/tree/master/sample-category/pod-delete"
community: "https://kubernetes.slack.com/messages/CNXNB0ZTN"
description: "kills nginx pods in a random manner"
keywords:
  - "pods"
  - "kubernetes"
  - "sample-category"
  - "nginx"
scope: "Namespaced"
auxiliaryappcheck: false
permissions:
  - apigroups:
      - ""
      - "batch"
      - "litmuschaos.io"
    resources:
      - "jobs"
      - "pods"
      - "chaosengines"
      - "chaosexperiments"
      - "chaosresults"
    verbs:
      - "create"
      - "list"
      - "get"
      - "update"
      - "patch"
      - "delete"
maturity: "alpha"
maintainers:
  - name: "ispeakc0de"
    email: "shubham.chaudhary@harness.io"
provider:
  name: "Harness"
minkubernetesversion: "1.12.0"
references:
  - name: Documentation
    url: "https://docs.litmuschaos.io/docs/getstarted/"

• Run the following command to generate the necessary artifacts for submitting the sample-category chaos chart with pod-delete experiment.

## litmus-go
$ ./litmus-sdk generate <generate-type> -f=attributes.yaml

• Note: Replace the <generate-type>placeholder with the appropriate value based on the usecase:
  • experiment: Chaos experiment artifacts belonging to a an existing OR new chart.
    • Provide the type of chaoslib in the -t flag. It supports the following values
      • exec: It creates the exec based chaoslib(default type)
      • helper: It creates the helper based chaoslib
  • chart: Just the chaos-chart metadata, i.e., chartserviceversion yaml
    • Provide the type of chart in the -t flag. It supports the following values
      • category: It creates the chart metadata for the category i.e chartserviceversion, package manifests
      • experiment: It creates the chart for the experiment i.e chartserviceversion, engine, rbac, experiment manifests
      • all: it creates both category and experiment charts (default type)
  • provide the path of the attribute.yaml manifest in the -f flag.

## litmus-ansible
$ python3 generate_chart.py --attributes_file=attributes.yaml --generate_type=experiment

• Note: In the --generate_type flag, select the appropriate type of manifests to be generated, where,
  • chart: Just the chaos-chart metadata, i.e, chartserviceversion yaml
  • experiment: Chaos experiment artifacts belonging to an existing or new chart.
  • provide the path of the attribute.yaml manifest in the --attributes_file flag.

Verify the Generated Files

$ cd /experiments

$ ls -ltr

total 8
drwxr-xr-x 3 shubham shubham 4096 May 15 12:02 generic/
drwxr-xr-x 3 shubham shubham 4096 May 15 13:26 sample-category/


$ ls -ltr sample-category/

total 12
-rw-r--r-- 1 shubham shubham   41 May 15 13:26 sample-category.package.yaml
-rw-r--r-- 1 shubham shubham  734 May 15 13:26 sample-category.chartserviceversion.yaml
drwxr-xr-x 2 shubham shubham 4096 May 15 13:26 sample-pod-delete/

$ ls -ltr sample-category/sample-pod-delete

total 28
-rw-r--r-- 1 shubham shubham  791 May 15 13:26 rbac.yaml
-rw-r--r-- 1 shubham shubham  734 May 15 13:26 sample-pod-delete.chartserviceversion.yaml
-rw-r--r-- 1 shubham shubham  792 May 15 13:26 experiment.yaml
-rw-r--r-- 1 shubham shubham  813 May 15 13:26 engine.yaml
## this file will be created in case of litmus-go
--rw-r--r-- 1 shubham shubham 4096 May 15 13:26 test
-rw-r--r-- 1 shubham shubham 4533 May 15 13:26 sample-pod-delete.go
## this file will be created in case of litmus-ansible
-rw-r--r-- 1 shubham shubham 1777 May 15 13:26 pod-delete-k8s-job.yml
-rw-r--r-- 1 shubham shubham 4533 May 15 13:26 pod-delete-ansible-logic.yml

$ ls -ltr sample-category/sample-pod-delete/test

total 4
-rw-r--r-- 1 shubham shubham  1039 May 15 13:26 test.yaml

• Proceed with construction of business logic inside the pod-delete.go file in litmus-go or pod-delete-ansible-logic.yml in litmus-ansible, by making the appropriate modifications listed below to achieve the desired effect:

  • variables
  • entry & exit criteria checks for the experiment
  • helper utils in either pkg or new base chaos libraries

• Update the experiment.yaml with the right chaos params in the spec.definition.env with their default values.

• Update the chaoslib/litmus/sample-pod-delete/lib/pod-delete.go chaoslib to achieve the desired effect or reuse the existing chaoslib.

• Create an experiment README explaining, briefly, the what, why & how of the experiment to aid users of this experiment.

Steps to Test Experiment

Follow the Litmus SDK: DevTest your chaos experiments with Okteto blog to test the generated experiment.

Steps to Include the Chaos Charts/Experiments into the ChartHub

• Send a PR to the litmus-go/litmus-go repo with the modified experiment files i.e, pod-memory-hog
• Send a PR to the chaos-charts repo with the modified experiment CR, experiment chartserviceversion, chaos chart (category-level) chartserviceversion & package (if applicable) YAMLs i.e, kubelet service kill

Conclusion

Litmus SDK will help developers & SREs to create their custom chaos experiment on demand. It will generate the templates, pre/post chaos checks, chaoslib (which can be modified according to use-case).

Are you an SRE or a Kubernetes enthusiast? Does Chaos Engineering excite you? Join Our Community #litmus channel in Kubernetes Slack
Contribute to LitmusChaos and share your feedback on Github
If you like LitmusChaos, become one of the many stargazers here