Ajeet Singh Raina

Posted on Nov 10

Taking Docker Compose to Production with One Tool

#docker #microservices #containers #kubernetes

Imagine you're using Docker Compose to develop and test your app. It’s a great tool for getting multiple containers to work together on a single host, which is perfect for local development. But when it comes to production, especially if you need to scale or handle higher traffic, Kubernetes often makes more sense.

But why?

Because Kubernetes adds built-in support for things like load balancing, service discovery, and automatic failover, which are essential for managing complex, distributed applications in production.

That’s where tools like Compose Bridge come in. You can develop with Docker Compose and then, when you’re ready to go live, convert your Compose files to Kubernetes resources. This way, you’re not sacrificing the simplicity of Compose during development but still getting the power of Kubernetes when you need it in production. It’s a smooth way to bridge the gap between development and scalable production.

What is Compose Bridge?

Compose Bridge, an experimental feature in Docker Desktop, offers a powerful solution for transforming Docker Compose configurations into Kubernetes manifests, bridging the gap between local development and production-ready orchestration. This tool significantly simplifies the transition from Docker Compose to Kubernetes, allowing developers to leverage the simplicity of Compose while harnessing the scalability and robustness of Kubernetes deployments.

Understanding Compose Bridge

At its core, Compose Bridge operates by utilizing transformations packaged as Docker images. These transformations receive a fully resolved Compose model as input (/in/compose.yaml) and produce target format files as output (/out). The default transformation provided by Compose Bridge focuses on generating Kubernetes manifests and a Kustomize overlay, specifically designed for deployment on Docker Desktop with Kubernetes enabled.

The transformation process leverages Go templates, which allows for easy customization and extension. This templating system enables developers to insert logic and data dynamically, making the generated Kubernetes manifests adaptable to various project requirements. The use of Go templates also provides access to a set of YAML helper functions, such as seconds for converting durations, uppercase for string manipulation, and base64 for encoding secrets.

Key Benefits

Key benefits of using Compose Bridge include:

Simplified Migration: It significantly reduces the complexity of transitioning from Docker Compose to Kubernetes, allowing developers to maintain the simplicity of Compose while leveraging Kubernetes' orchestration capabilities.
Customizability: The templating system allows for extensive customization, enabling organizations to tailor the transformation process to their specific infrastructure needs and preferences.
Consistency: By automating the conversion process, Compose Bridge helps maintain consistency between development and production environments, reducing potential discrepancies and deployment issues.
Resource Optimization: The generated Kubernetes manifests include advanced features like resource limits, health checks, and network policies, which can lead to more efficient resource utilization in Kubernetes clusters.
Rapid Prototyping: Developers can quickly prototype and test Kubernetes deployments based on their existing Docker Compose files, accelerating the development and deployment cycle.
Learning Tool: For teams transitioning to Kubernetes, Compose Bridge serves as an educational tool, providing insights into how Docker Compose concepts map to Kubernetes resources.

While Compose Bridge offers significant advantages, it's important to note that as an experimental feature, it may undergo changes in future releases. Additionally, complex applications might still require manual adjustments to fully optimize for Kubernetes deployments. Despite these considerations, Compose Bridge represents a valuable tool in the Docker ecosystem, streamlining the containerization workflow from development to production.

How is it different from other conversion tools?

Great question.

Integration with Docker Desktop - Compose Bridge is designed to work seamlessly with Docker Desktop, providing a default transformation that generates Kubernetes manifests and a Kustomize overlay specifically for deployment on Docker Desktop with Kubernetes enabled. Learn More
kubectl Plugin Functionality - Compose Bridge can function as a kubectl plugin, allowing for direct integration into Kubernetes command-line operations. Learn More
Go Template-Based Transformation: Compose Bridge uses Go templates for its default Kubernetes transformation, making it easy to extend and customize. Learn More
Flexibility and Customization: Compose Bridge allows for extensive customization of the transformation process. Users can modify existing templates or add their own to generate Kubernetes manifests that meet specific needs. Learn More
Built-in Helper Functions: Compose Bridge offers a set of YAML helper functions designed to manipulate data within the templates efficiently. Learn More

Setting Up Compose Bridge

Prerequisites

Before setting up Compose Bridge, ensure:

Download and Install Docker Desktop
Enabled Kubernetes in Docker Desktop

Steps

Sign in to your Docker account within Docker Desktop
Navigate to the "Features in development" tab in Settings
Select the "Experimental features" tab
Enable the "Compose Bridge" option

After enabling the feature, Compose Bridge becomes available for use.

Overview of the Default Kubernetes Transformation

The default transformation simplifies initial migrations, converting Compose files to Kubernetes-ready configurations. This transformation is intended to quickly generate Kubernetes YAML manifests that map closely to the original Docker Compose settings.

How it works

Compose bridge uses transformations to let you convert a Compose model into another form. A transformation is packaged as a Docker image that receives the fully-resolved Compose model as /in/compose.yaml and can produce any target format file under /out. Compose Bridge provides its transformation for Kubernetes using Go templates, so that it is easy to extend for customization by just replacing or appending your own templates.

The default Compose Bridge transformation generates Kubernetes resources that include:

Pods: representing containers for each service.
Services: creating a network interface for each container.
ConfigMaps and Secrets: managing environment variables and sensitive data.

How to Use the Default Transformation

Navigate to your project directory containing the Compose file.
Run the command:

$ compose-bridge convert --help

$ compose-bridge convert --help

Usage:  docker compose-bridge convert [OPTIONS]

Convert a Compose file to Kubernetes resources

Options:
  -f, --file stringArray             Compose configuration files
  -o, --output string                The output directory for the Kubernetes resources (default "out")
      --templates string             Directory containing transformation templates
  -t, --transformation stringArray   Transformation to apply to compose model (default:
                                     docker/compose-bridge-kubernetes)

If you want to convert a compose.yaml file that is located in another directory, you can run:

compose-bridge convert -f <path-to-file>/compose.yaml

Compose Bridge will generate and apply Kubernetes manifests based on the Compose file, deploying the application on your Kubernetes cluster.

Learn more about the default transformation on the official Docker docs

Sample Application: From Compose to Kubernetes

We’ll use a basic example featuring a web server and database service for this demonstration.

Writing the Docker Compose File

Create a docker-compose.yml with the following configuration:

services:
  web:
    image: nginx
    ports:
      - "80:80"
  db:
    image: postgres
    environment:
      POSTGRES_PASSWORD: example

Using Compose Bridge to Transform the Compose File

Compose Bridge will convert docker-compose.yml into Kubernetes manifests and apply them.

compose-bridge convert
latest: Pulling from library/postgres
5ecbc93de950: Download complete
ba6e84deecdb: Download complete
425a818d8223: Download complete
159c4494c279: Download complete
b0dd2bf7a180: Download complete
f7fdac227b96: Download complete
3dc01759541c: Download complete
c0a40d31bfac: Download complete
90646cadd7e6: Download complete
e868621ffb41: Download complete
f6ce5b6f352e: Download complete
e1dbef7dcaa1: Download complete
6b732c95d024: Download complete
Digest: sha256:8d3be35b184e70d81e54cbcbd3df3c0b47f37d06482c0dd1c140db5dbcc6a808
Status: Downloaded newer image for postgres:latest
latest: Pulling from library/nginx
2c1384c86539: Download complete
51635e63ab0c: Download complete
805908969407: Download complete
b9a670e7a7f3: Download complete
6c29a458e7d5: Download complete
1f62b39dc401: Download complete
Digest: sha256:28402db69fec7c17e179ea87882667f1e054391138f77ffaf0c3eb388efc3ffb
Status: Downloaded newer image for nginx:latest
latest: Pulling from docker/compose-bridge-kubernetes
af80f65e8b88: Download complete
062101437435: Download complete
99d8bc75d872: Download complete
Digest: sha256:698ecbaaf8abf299cddcf1f9de08509858a9e9199cc6d2f19ba5bf650b72c334
Status: Downloaded newer image for docker/compose-bridge-kubernetes:latest
Kubernetes resource db-deployment.yaml created
Kubernetes resource web-deployment.yaml created
Kubernetes resource db-expose.yaml created
Kubernetes resource web-expose.yaml created
Kubernetes resource 0-10nov-namespace.yaml created
Kubernetes resource default-network-policy.yaml created
Kubernetes resource web-service.yaml created
Kubernetes resource kustomization.yaml created
Kubernetes resource web-service.yaml created
Kubernetes resource kustomization.yaml created

Analyzing the Generated Kubernetes Manifests

The above command creates manifests that represent each service as a Kubernetes pod. These files are then stored within your project in the /out folder.

tree out
out
├── base
│   ├── 0-10nov-namespace.yaml
│   ├── db-deployment.yaml
│   ├── db-expose.yaml
│   ├── default-network-policy.yaml
│   ├── kustomization.yaml
│   ├── web-deployment.yaml
│   ├── web-expose.yaml
│   └── web-service.yaml
└── overlays
    └── desktop
        ├── kustomization.yaml
        └── web-service.yaml

4 directories, 10 files

The Kubernetes manifests can then be used to run the application on Kubernetes using the standard deployment command kubectl apply -k out/overlays/desktop/.

kubectl apply -k out/overlays/desktop/
namespace/10nov created
service/db created
service/web created
service/web-published created
deployment.apps/db created
deployment.apps/web created
networkpolicy.networking.k8s.io/default-network-policy created

You can verify this by running the CLI too:

kubectl get pods -A
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
10nov         db-7c7d8ffc7f-c6k2p                      1/1     Running   0          27s
10nov         web-b7c76985f-bn5lc                      1/1     Running   0          27s
kube-system   coredns-7db6d8ff4d-5g8ph                 1/1     Running   0          4m20s
kube-system   coredns-7db6d8ff4d-vn95x                 1/1     Running   0          4m20s
kube-system   etcd-docker-desktop                      1/1     Running   2          4m26s
kube-system   kube-apiserver-docker-desktop            1/1     Running   2          4m21s
kube-system   kube-controller-manager-docker-desktop   1/1     Running   2          4m17s
kube-system   kube-proxy-bt6pf                         1/1     Running   0          4m21s
kube-system   kube-scheduler-docker-desktop            1/1     Running   2          4m19s
kube-system   storage-provisioner                      1/1     Running   0          4m18s
kube-system   vpnkit-controller                        1/1     Running   0          4m18s

Examining the YAML files

cd out/overlays/desktop/

cat kustomization.yaml
#! kustomization.yaml
# Generated code, do not edit
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
    - ../../base
patches:
    - path: web-service.yaml

cat web-service.yaml
# check if there is at least one published port

#! web-service.yaml
# Generated code, do not edit
apiVersion: v1
kind: Service
metadata:
    name: web-published
    namespace: 10nov
spec:
    type: LoadBalancer

Now you can access Nginx running inside a Pod by accessing the web browser:

You can verify if Postgres is running or not by examining it's logs:

Volumes:
  kube-api-access-rb8zn:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Scheduled       9h    default-scheduler  Successfully assigned 10nov/db-7c7d8ffc7f-c6k2p to docker-desktop
  Normal  Pulled          9h    kubelet            Container image "postgres" already present on machine
  Normal  Created         9h    kubelet            Created container db
  Normal  Started         9h    kubelet            Started container db
  Normal  SandboxChanged  21m   kubelet            Pod sandbox changed, it will be killed and re-created.
  Normal  Pulled          21m   kubelet            Container image "postgres" already present on machine
  Normal  Created         21m   kubelet            Created container db
  Normal  Started         21m   kubelet            Started container db

Resources

Top comments (9)

Benoit COUETIL 💫 • Nov 11

Thank you for sharing.

How does it compare to Kompose ? How do you handle needed configuration details that are obviously not part of the docker compose file definition ?

Ajeet Singh Raina • Nov 12 • Edited

Great question.

I have used Kompose in the past. I wrote a blog post collabnix.com/translate-a-docker-c... last Nov. TBH, Kompose is an opinonated transformation. If you don't like the way it translates compose stuff into k8s (labels, annotations, ports?) you're stuck.

Compose Bridge uses Go templates for transformations, making it easy to extend or customize. You can learn more about it here: docs.docker.com/compose/bridge/cus...

Ajeet Singh Raina • Nov 12

Let me answer your second question.

In Docker Compose, you can define a service and expose its ports, but there’s no native support for things like Kubernetes Ingress (which allows external HTTP access to your service). So, if you're using Compose-bridge, you might want to add custom metadata (such as x-ingress) to your Compose file so that Compose-bridge knows to generate an Ingress resource when transforming the Compose file into Kubernetes YAML.

Here's an example of a Compose file with a service definition that includes the x-ingress custom attribute:

services:
  test:
    ports:
      - target: 80
        x-ingress: /test

Now when you run Compose-bridge with this Compose file, it looks at the custom attribute x-ingress: /test and knows that you want to create a corresponding Kubernetes Ingress resource for your service. It then generates a Kubernetes YAML that includes an Ingress definition like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
spec:
  rules:
  - http:
      paths:
      - path: /test
        pathType: Prefix
        backend:
          service:
            name: test
            port:
              number: 80

What's Happening in the Kubernetes YAML?

Ingress: This is a Kubernetes resource that allows external traffic to reach your service. path: /test: This defines the URL path that external traffic will use to reach the test service. backend: This points to the test service on port 80.

Why This is Useful:

Customization: Compose-bridge allows you to customize your Compose file with extra attributes (like x-ingress) that it uses during the transformation to Kubernetes resources.
Flexible Generation: This method enables the automatic generation of Kubernetes-specific resources (like Ingress) that aren’t directly available in Docker Compose.

In nutshell,

By adding x-ingress: /test in the Compose file, you're providing additional information that Compose-bridge uses to generate the appropriate Kubernetes Ingress resource. This is a way to bridge the gap between Docker Compose and Kubernetes, customizing the transformation for your specific needs.