Develop Right on Top of Your Kubernetes Cluster

In recent years, Kubernetes has emerged as the de facto standard for container orchestration, providing an efficient and scalable platform for deploying and managing applications. As developers strive to streamline their workflows and optimize productivity, an emerging trend is developing applications directly on top of the Kubernetes cluster itself. In this blog post, we will explore the benefits of developing right on top of your Kubernetes cluster, and dive into the challenges that make this a difficult reality to achieve.

Starting with the benefits -

Reduced env variance

By developing directly on your Kubernetes cluster, you eliminate the need for a separate development environment or local setup. This streamlines the development process, reducing the overhead of replicating the cluster setup locally, and therefore the time spent on setting up and maintaining multiple environments.

In a complex project there may be multiple flavors of production environments along with a host of staging, test, and preview environments. Having to maintain yet another environment type (which is used by the entire dev team!) is a huge time and attention sink for both developers and devops.

Consistent development and production envs

Catch and resolve issues early on, and reduce the risk of surprises during deployment by using the same tools, libraries, and configurations as the production environment. Developing on top of your Kubernetes cluster ensures that your development environment is aligned with the production environment. This consistency eliminates the "works on my machine" problem, where code behaves differently in different environments.

Seamless integration with cluster services

Developing on top of your Kubernetes cluster gives you direct access to cluster services and features. You can easily integrate with monitoring tools, logging systems, service meshes, and other cluster-specific resources. This allows you to develop applications that are tightly coupled with the underlying infrastructure, leveraging the full capabilities of Kubernetes and maximizing the efficiency of your applications.

Improved collaboration and teamwork

Kubernetes provides powerful features for collaboration, allowing multiple developers to work simultaneously on the same cluster. With development happening directly on the cluster, team members can easily share links to environments (using Kubernetes Ingress objects). This enhances teamwork and promotes knowledge sharing, as developers can observe and learn from each other's work.

Simplified testing

Developing directly on your Kubernetes cluster simplifies the testing process. Since you are working on the same environment where your application will run, you can easily reproduce and analyze issues that might arise during deployment. This reduces the time spent on recreating issues locally and provides a more accurate representation of how the application behaves in the production environment.

So why isn’t everyone developing on top of their Kubernetes cluster?

Unfortunately, there are a couple of significant challenges that make developing on top of a cluster difficult -

Container image immutability

Containers are designed to be immutable, meaning they can not be modified once built. While this immutability promotes scalability and consistency, it poses a big challenge for developers - it is no longer possible to make code changes and see fast feedback! Instead, developers rely on their CI pipeline to deploy new images, which extends the feedback cycle from seconds to 30+ minutes.

Long cycles leads to bundling many changes together, which in turn causes each cycle’s success rate to drop, and inevitably dev speed is slowed significantly.

Inability to debug

To make matters worse, it is difficult to set up proper debugging for code running in a Kubernetes cluster.

• The container image does not contain the required debugging toolchain (debugpy for Python, dlv for Go, etc.).
• The container may not have the required security configuration (read only root FS / missing capabilities / …)
• Ports need to be opened between the IDE and the container running in the cluster.
• The IDE needs to be configured with the correct debugging configuration.

All of these mean that practically - developers cease debugging once their env is running on Kubernetes.

Complexity

Kubernetes is a highly sophisticated and complex system. While some developers have a strong grasp of the basic Kubernetes concepts, the learning curve can be steep and it can slow down developers significantly.

Lack of access

In most organizations, access to Kubernetes clusters (even those intended for development) is not available to all developers. For good reason too - it is easy to make mistakes and cause damage that takes time and resources to repair, and the teams maintaining the clusters are wise to take proactive measures to minimize the chance of that. The larger the organization, and the more complex the deployment, the higher the chance of limitations being in place for its clusters.

Cost and environment management

Managing the cost of a Kubernetes cluster is an art in itself. Significantly increasing the number of environments due to their use in development can increase costs significantly. Managing the large number of dev environments can add a lot of overhead to already overextended DevOps or platform teams. Existing tooling intended to provide visibility into a limited number of environments may not scale well to the higher usage. And differences in configuration between environment types can cause overhead and false-positives.

Wait - how is anyone developing on their Kubernetes cluster?

Luckily, there are all kinds of tools that can be used to manage and develop on dev environments on Kubernetes. Everything from Spinnaker to ArgoCD and FluxCD for orchestration, and various solutions for the developer experience layer.

We use our own product (Raftt), which handles all of this complexity [and more]. Raftt adapts the Kubernetes environment for development, enabling out of the box hot-reloading of code and debugging, connected directly to your local machine and IDE. Raftt’s optional cluster-level controller handles environment provisioning and lifecycle, minimizing maintenance and additional costs.

You are left with a great development experience (including code hot reloading, debugging, collaboration…) that is easy to manage and deploy. You can try it yourself - deploy Raftt for development on your Kubernetes cluster, with our demo project or with your own code.