Chaos Engineering Concepts

Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production.

Advances in large-scale, distributed Software systems are changing everything related to Software Engineering. Flexibility of development and velocity of deployment are good examples, which show the change principles.

The good point about chaos process is: How much confidence we can have in the complex systems that are put in a production environment.

Resilience patterns improve any mechanism for distributed systems to be reliable, create scenarios of errors, identify weakness and aberrant behaviors. Systemic weakness could take the form of:

• Improper fallback settings when service is unavailable

• Retry storms from improperly tuned timeouts

• Outages when a downstream dependency receives too much traffic

• Cascading failures when a single point of failure crashes

For each error we have resilience patterns implementation in a github repository and using Java, Spring and Resilience4J.

We need a way to manage the chaos inherent in these systems, take advantage of increasing flexibility and velocity and have confidence in production deployments.

Chaos Engineering with Chaos Mesh

Chaos Mesh is an open source cloud-native Chaos Engineering platform. It offers various types of fault simulation and has an enormous capability to orchestrate fault scenarios. Using Chaos Mesh, you can conveniently simulate various abnormalities that might occur in reality during the development, testing, and production environments and find potential problems in the system. To lower the threshold for a Chaos Engineering project, Chaos Mesh provides you with a perfect visualization operation. You can easily design your Chaos scenarios on the Web UI and monitor the status of Chaos experiments.

Chaos Mesh Architecture

Chaos Mesh is built on Kubernetes CRD (Custom Resource Definition). To manage different Chaos experiments, Chaos Mesh defines multiple CRD types based on different fault types and implements separate Controllers for different CRD objects. Chaos Mesh primarily contains three components:

• Chaos Dashboard: The visualization component of Chaos Mesh. Chaos Dashboard offers a set of user-friendly web interfaces through which users can manipulate and observe Chaos experiments. At the same time, Chaos Dashboard also provides an RBAC permission management mechanism.

• Chaos Controller Manager: The core logical component of Chaos Mesh. The Chaos Controller Manager is primarily responsible for the scheduling and management of Chaos experiments. This component contains several CRD Controllers, such as Workflow Controller, Scheduler Controller, and Controllers of various fault types.

• Chaos Daemon: The main executive component. Chaos Daemon runs in the DaemonSet mode and has the Privileged permission by default (which can be disabled). This component mainly interferes with specific network devices, file systems, and kernels by hacking into the target Pod Namespace.

Chaos Mesh Faults

1. Basic resource faults

1.1. PodChaos: simulates Pod failures, such as Pod node
restart, Pod's persistent unavailability, and certain
container failures in a specific Pod.

1.2. NetworkChaos: simulates network failures, such as
network latency, packet loss, packet disorder, and
network partitions.

1.3. DNSChaos: simulates DNS failures, such as the parsing
failure of DNS domain name and the wrong IP address
returned.

1.4. HTTPChaos: simulates HTTP communication failures, such
as HTTP communication latency.

1.5. StressChaos: simulates CPU race or memory race.

1.6. IOChaos: simulates the I/O failure of an application
file, such as I/O delays, read and write failures.

1.7. TimeChaos: simulates the time jump exception.

1.8. KernelChaos: simulates kernel failures, such as an
exception of the application memory allocation.

2. Platform faults:

2.1. AWSChaos: simulates AWS platform failures, such as the
AWS node restart.

2.2. GCPChaos: simulates GCP platform failures, such as the
GCP node restart.

3. Application faults:

3.1. JVMChaos: simulates JVM application failures, such as
the function call delay.

Installing Chaos Mesh in Kubernetes Cluster

Beginning implementation of chaos mesh on kubernetes we used this Documentation or clone this repository on branch chaos-mesh

Now with this repository, we need to have installed in your machine docker and minikube. After this process you run this command to start the Kubernetes cluster.

minikube start

Now Installing Chaos Mesh:

#install chaos mesh
curl -sSL https://mirrors.chaos-mesh.org/v2.1.5/install.sh | bash

# check pods
kubectl get po -n chaos-testing

>> output
NAME                                       READY   STATUS    RESTARTS      AGE
chaos-controller-manager-bc4b9bf7b-2jd8m   1/1     Running   2 (15m ago)   7d3h
chaos-controller-manager-bc4b9bf7b-8qrnq   1/1     Running   2 (15m ago)   7d3h
chaos-controller-manager-bc4b9bf7b-pcxzc   1/1     Running   1 (15m ago)   7d3h
chaos-daemon-qmk4v                         1/1     Running   1 (15m ago)   7d3h
chaos-dashboard-599fc7bdf4-hf2sc           1/1     Running   2 (15m ago)   7d3h

# check services
kubectl get svc -n chaos-testing

>> output 
NAME                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                 AGE
chaos-daemon                    ClusterIP   None            <none>        31767/TCP,31766/TCP                     7d3h
chaos-dashboard                 NodePort    10.103.107.62   <none>        2333:30176/TCP,2334:31739/TCP           7d3h
chaos-mesh-controller-manager   ClusterIP   10.98.129.185   <none>        443/TCP,10081/TCP,10082/TCP,10080/TCP   7d3h

# access chaos mesh dashboard via NodePort service
# dashboard run on port 2333 as NodePort service
# dashboard can access via NodePort port 2333 mapping port, in my case port 31725
http://<minikube ip>:30176
http://192.168.49.2:30176/dashboard

# access dashboard via port forward 8888 -> 2333
# now the dashboard can be accessed via localhost:8888
kubectl port-forward -n chaos-testing chaos-dashboard-599fc7bdf4-hf2sc 8888:2333

http://localhost:8888

To finish the whole process, you accessChaos Mesh Dashboard

Chaos Mesh Scenarios

There are two ways to run Chaos Mesh experiments. One method is directly creating experiments via the Chaos Mesh web dashboard. Another method is .yaml file deployments. If you need more details access Chaos Mesh Documentation

Api User Deployment.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: chaos

---

apiVersion: v1
kind: Service
metadata:
  name: user-api
  labels:
    name: user-api
  namespace: chaos
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  type: NodePort
  selector:
    app: user-api

---

apiVersion: v1
kind: ReplicationController
metadata:
  name: user-api
  namespace: chaos
spec:
  replicas: 3
  selector:
    app: user-api
  template:
    metadata:
      name: user-api
      labels:
        app: user-api
    spec:
      containers:
        - name: user-api
          image: lucasnscr/user-api:0.0.1
          ports:
            - containerPort: 8080

Now, we have the deployment and service created. Go create chaos for Api User, we are going create scenario pod failure:

kind: PodChaos
apiVersion: chaos-mesh.org/v1alpha1
metadata:
  name: chaos-failure
  namespace: chaos
spec:
  selector:
    namespaces:
      - chaos
    labelSelectors:
      app: user-api
  mode: one
  action: pod-failure
  duration: 30s
  gracePeriod: 0

Create another scenario of chaos, we now pod kill:

kind: PodChaos
apiVersion: chaos-mesh.org/v1alpha1
metadata:
  name: chaos-kill
  namespace: chaos
spec:
  selector:
    namespaces:
      - chaos
    labelSelectors:
      app: user-api
  mode: one
  action: pod-kill
  duration: 30s
  gracePeriod: 0

Now we created a stress test for Api User pod:

kind: StressChaos
apiVersion: chaos-mesh.org/v1alpha1
metadata:
  name: chaos-stress
  namespace: chaos
spec:
  selector:
    namespaces:
      - chaos
    labelSelectors:
      app: user-api
  mode: one
  containerNames:
    - ''
  stressors:
    memory:
      workers: 4
      size: '256MB'
  duration: 60s

Chaos Mesh improves any situation, is a good tool for the Kubernetes environment and creates specific situations of chaos.

Chaos Mesh is a project adopted by the Cloud Native Computing Foundation.