Case study: How blue-green deployment helped Mochi

At Dwarves Foundation, our team has always faced some bit of friction when deploying apps for our clients. We’ve known about blue-green deployments for a while and were recently given the chance to evaluate and demonstrate them for one of our Discord bot projects.

Applying Blue-green deployment for Mochi Bot

Introducing Mochi Bot to the Web3 space, our team has developed a flexible and user-friendly product with features like NFT rarity queries, sales alerts on Discord and Twitter, and showing various tips. To enhance user experience and streamline deployment, we had the chance to implement blue-green deployment for the Mochi Bot application. Below is our case study that evaluates the cost and practicality of this deployment strategy in our current infrastructure.

Current Infrastructure & Implementation Plan

Mochi Bot runs on a Kubernetes infrastructure managed by ArgoCD, with two pods. By implementing blue-green deployment, we aim to eliminate downtime and ensure consistent updates across pods. To achieve this, we set up two identical production environments (blue and green) and followed these steps:

1. Evaluate current infrastructure and identify prerequisites.
2. Prepare Kubernetes manifests for blue and green environments.
3. Apply new configurations to clusters.
4. Test deployment process, including traffic switching and rollback procedures.

Preparation & Resource Definition

To implement blue-green deployment in Kubernetes, we needed to define the application resources and the rollout strategy. We defined the application resources using three YAML files:

.
└── app/
    ├── bluegreen-rollout.yaml
    ├── ingress.yaml
    └── service.yaml

Before that, setting up Argo Rollouts is a prerequisite to enable blue-green deployment capability:

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

The blue-green update strategy is applied to define the release through the bluegreen-rollout.yaml file.

# bluegreen-rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: mochiapp
  labels:
    app: mochiapp
spec:
  replicas: 1
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app: mochiapp
  template:
    metadata:
      labels:
        app: mochiapp
    spec:
      containers:
        - name: myapp
          image: argoproj/rollouts:blue
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 8080
  strategy:
    blueGreen:
      autoPromotionEnabled: false
      activeService: mochiapp
      previewService: mochiapp-preview

The bluegreen-rollout.yaml file is structured similarly to a typical deployment file, but with the added strategy section. Here, we specified the activeService to update with the new template hash during promotion (required), and the previewService to update with the new template hash before (optional).

Next, we created the ingress.yaml and service.yaml files, which contained the respective configurations for ingress and service resources:

# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mochiapp
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - mochiapp.bluegreen.domain
      secretName: mochiapp.bluegreen.domain-tls
  rules:
    - host: mochiapp.bluegreen.xyz
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: mochiapp
                port:
                  name: http

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mochiapp-preview
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - preview.mochiapp.xyz
      secretName: preview-mochiapp.bluegreen.domain-tls
  rules:
    - host: preview-mochiapp.bluegreen.domain
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: mochiapp-preview
                port:
                  name: http

# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: mochiapp
  labels:
    app: mochiapp
spec:
  selector:
    app: mochiapp
  ports:
    - name: http
      port: 80
      targetPort: http
  type: ClusterIP

---
apiVersion: v1
kind: Service
metadata:
  name: mochiapp-preview
  labels:
    app: mochiapp
spec:
  selector:
    app: mochiapp
  ports:
    - name: http
      port: 80
      targetPort: http
  type: ClusterIP

Applying Resources to the Cluster and Testing

We applied the resource files to the Kubernetes cluster using kubectl apply commands. The application was deployed to the active environment and could be viewed on the production domain - mochiapp.bluegreen.domain.

kubectl apply -f app/bluegreen-rollout.yaml
kubectl apply -f app/service.yaml
kubectl apply -f app/ingress.yaml

When releasing a new version, we changed the image of the application and applied the changes to the cluster. Argo rollouts created a new ReplicaSet with the new image and initiated the rollout of the updated version in the preview environment, accessible at the preview domain - preview-mochiapp.bluegreen.domain.

# bluegreen-rollout.yaml
containers:
    - name: myapp
      image: argoproj/rollouts-demo:green

kubectl apply -f app/bluegreen-rollout.yaml

Once the updated application version was thoroughly tested, we promoted it to the active environment using a specific command. The updated application was then accessible on the active environment in the blue domain.

kubectl argo rollouts promote mochi

With the deployment complete, the updated application was accessible on the active environment in the blue domain - mochiapp.bluegreen.domain.

An alternative would be to utilize the ArgoCD UI for the promotion, as it could prove to be exceptionally helpful for those who may not be able to operate the CLI, including QC, during the release rollout. Additionally, any issues that may arise can quickly be reverted with just the click of the "Rollback" button.

ArgoCD rollout dashboard to manage blue-green applications

ArgoCD service Mochi API

Conclusion

This case study helped us demonstrate the value of blue-green deployment in reducing downtime, improving user experience, and streamlining the update process for applications like Mochi Bot.

Implementing a blue-green deployment strategy for Mochi Bot proved to be a smooth and hassle-free process. It provided an immediate benefit by eliminating inconsistencies between pod updates during deployment. Thankfully, as a result, the user experience remained seamless and uninterrupted during application updates.

Moving forward, our team plans to integrate K6 for API testing to improve the performance and reliability of Mochi Bot. We aim to establish a quality gate for the green version to ensure that only thoroughly tested and stable releases are deployed to the live environment.

Follow us on

Website: https://dwarves.foundation
Discord: https://discord.gg/dwarvesv
Fanpage: https://www.facebook.com/dwarvesf
LinkedIn: https://www.linkedin.com/company/dwarvesf
Substack: https://note.d.foundation/