Cloud Development Environments: Using Skaffold and Telepresence on Kubernetes for fast dev loops

Build and deploy microservices with Skaffold; Test against remote dependencies with Telepresence

Skaffold is an open source project created by Google. It provides a development framework for Kubernetes based applications. This framework creates a fast, repeatable, and simple local Kubernetes workflow. Put simply, this means that it handles all the hard bits of managing a cloud native local development environment, such as building images and deploying to Kubernetes. With Skaffold handling the hard bits, you save time with every iteration you make in development.

Telepresence is an open source tool created by Ambassador Labs. Telepresence makes Kubernetes developers super productive by letting them code as if their laptop is in their Kubernetes cluster. This way you can query cluster resources and experiment rapidly with other services in realtime.

Together, Skaffold and Telepresence give developers a supercharged development workflow for Kubernetes, where Skaffold handles building and deploying your local service and Telepresence allows you to test it against services in your remote cluster.

In this tutorial, we’ll use Skaffold to build and deploy our local environment. Then, we’ll spin up Telepresence to do some iterations while projecting the local service we are building to a remote cluster. Using Telepresence as part of Ambassador Cloud gives us some additional benefits, like being able to share preview URLs with teammates, so they can view the services running on your local machine as if they were in production. We’ll talk more about this in future blog posts.

CI/CD and the Inner Dev Loop

Continuous integration and continuous deployment — commonly referred to as CI/CD — is the de-facto method used to develop and deploy cloud native applications. On the right side of the diagram above, we have the continuous deployment workflow or outer loop. On the left-side of the diagram we have the development workflow, typically called the inner-dev-loop. This inner-dev-loop is where you write and test code before committing it to a version control system. Because you’re in the midst of the creative process of writing and testing code, you want this feedback loop to be as fast as possible. The faster this loop performs the faster you can refactor your code and test again.

The desire to make the loop faster is why tools like Skaffold and Telepresence, which are designed to save you time in this inner-dev-loop, are a great addition to Cloud Native workflows. With Skaffold managing your local deployments and watching your project directory as you make changes, you get a snappy feedback loop where your code appears in your local-dev-environment every time you save a file.

To make your inner-dev-loop even faster, Skaffold can be paired with Telepresence. With Skaffold managing your local service, and Telepresence safely intercepting traffic from your production cluster, you are able to test your code with nearly instantaneous feedback. On top of that, your code has been tested against real traffic saving you time later from debugging minor differences between dev and prod.

The Skaffold dev workflow

When using Skaffold to help manage your local environment you will find yourself in the Skaffold dev mode quite a bit. At the core of Skaffold dev there is a skaffold.yaml file. Sitting at the root directory of your project, this file contains all the info Skaffold needs to turn your code into containers and those containers into an application on your development cluster. Skaffold doesn’t do this alone though, Skaffold takes advantage of the popular tools you probably already use. Such as Dockerfiles or Webpacks to help build the containers, then Kubectl and Helm to deploy them. Because there are so many tooling options for Cloud Native applications, Skaffold has been designed to be highly modular to allow you to mix-and-match with tools you already use.

You can think of the Skaffold dev mode as a fully automated CI/CD pipeline for your development cluster. It takes all the tools you already use to deploy your application and does it automatically every time you make a change. This pipeline can get complex if needed, but at its primary levels it has 2 main stages: build and deploy. Let’s take a look at how this looks with the ‘react-reload’ example provided in the Skaffold repository.

Build

For those familiar with Docker, the default behaviour is to build a docker image using the provided Dockerfile. While this build stage can use other tools like JIB or Webpacks, the primary goal is to get your code packaged into a container so it is ready to be deployed.

From the example’s Dockerfile we see that it is creating a Node container with the port 8080 exposed. No special instructions are added for Skaffold, it is just a simple Dockerfile.

FROM node:12.16.0-alpine3.10
WORKDIR /app
EXPOSE 8080
CMD [“npm”, “run”, “dev”]
COPY package* ./
# examples don’t use package-lock.json to minimize updates
RUN npm install — no-package-lock
COPY . .

One of the most annoying things about managing containers is when you need to rebuild your images every time a file changes, even if the file doesn’t affect the state of the container. This is one area Skaffold really shines, because Skaffold dev manages the deployment, it can update files used by the container without having to rebuild the image.

However, if you make a change that does affect the container directly, Skaffold will pick up on the change and rebuild the necessary images. Then these newly created images are handed off to the deploy stage, where they are seamlessly integrated into your local dev environment. This seamless integration gives you a very fast turnaround time from code change to code tested.

Deploy

Now on to the Kubernetes bits! Skaffold is such a powerful tool because it allows Developers to focus on writing code and managing kubernetes. Switching between writing code and deploying it to your cluster causes an unnecessary context switch from Dev to Ops. In the deploy stage of the pipeline, Skaffold acts like your personal operations engineer. With all the code neatly packaged up from the build stage, Skaffold creates the deployments and services in your development cluster.

Still working with the ‘react-reload’ example, the deployment and service configurations for the sample application are defined in the manifest files located in k8s folder. Here we can see the react-reload image created in the build stage and service exposing port 8080. Skaffold also supports Helm and Kustomize manifest files.

apiVersion: v1
kind: Service
metadata:
  name: node
spec:
  ports:
  - port: 8080
  type: LoadBalancer
  selector:
    app: node
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: node
spec:
  selector:
    matchLabels:
      app: node
  template:
    metadata:
      labels:
        app: node
    spec:
      containers:
      - name: react
        image: react-reload
        ports:
        - containerPort: 8080

With these manifest files and some of the info in the skaffold.yaml file, Skaffold understands how to apply this configuration. With the default behaviour Skaffold will use kubectl as it is currently configured on your machine. Now that the sample application is up and running, Skaffold switches into watcher mode. Keeping the deployment and services up to date with the code in the project directory automagically. So now, as a developer you can go from writing code to testing code deployed to kubernetes without putting on your hat to be an operations engineer, saving you time and the mental tax from these context switches.

React-Reload with Skaffold Dev for fast build and deploy

Now to put all the pieces together and practice using Skaffold dev mode. I’ll walk you through deploying the ‘react-reload’ example to a local cluster running in Docker. Let’s start off by going through the prerequisites.

Prerequisites

1. Docker

2. Docker hub account (Necessary for Skaffold + Telepresence Demo)

3. Kubectl and Kubernetes cluster (running in docker)

4. Skaffold

Check requirements

1. docker version to check that docker is running

   $ docker version
   Client: Docker Engine — Community
   Cloud integration: 1.0.4
   ===trimmed for brevity===

   1. Checked that you are logged into docker

   $ docker login
   Authenticating with existing credentials…
   Login Succeeded

   1. kubectl get nodes should return docker-desktop

   $ kubectl get nodes
   NAME STATUS ROLES AGE VERSION
   docker-desktop Ready master 2d7h v1.19.3

   1. skaffold version to check that the Skaffold binary is available

   $ skaffold version
   v1.19.0

Example Steps

1. Clone Repo

If you haven’t already done so, clone the react reload example locally and cd into the read-reload directory.

$ git clone git@github.com:GoogleContainerTools/skaffold.git
$ cd skaffold/examples/react-reload

2. Start Skaffold Dev

Run skaffold dev — default-repo <repository-name> in the project directory to kick things off. Remember to replace the repository name with your personal repository.

*Note: We are using an external repository here. When using a local cluster skaffold dev works without an external repository, but we will be using this repository in a later section of the tutorial.

$ skaffold dev — default-repo peteroneilljr
Listing files to watch…
- react-reload
Generating tags…
- react-reload -> peteroneilljr/react-reload:v1.17.1–57-g95092addf-dirty
Checking cache…
- react-reload: Not found. Building
Found [docker-desktop] context, using local docker daemon.
Building [react-reload]…
Sending build context to Docker daemon 11.26kB
Step 1/7 : FROM node:12.16.0-alpine3.10
== trimmed for brevity ===
Step 7/7 : COPY . .
— -> 11fd5f0fcad4
Successfully built 11fd5f0fcad4
Successfully tagged peteroneilljr/react-reload:v1.17.1–57-
95092addf-dirty
Tags used in deployment:
- react-reload -> peteroneilljr/react-reload:11fd5f0fcad4559dbc176cabd46b91a01a82d3dd2b6604f7702f338e6275015e
Starting deploy…
- service/node created
- deployment.apps/node created
Waiting for deployments to stabilize…
- deployment/node is ready.
Deployments stabilized in 2.229471962s
Press Ctrl+C to exit
Watching for changes…
[react]
[react] > react-reload@1.0.0 dev /app
[react] > webpack-dev-server — mode development — hot
[react]
[react] ℹ ｢wds｣: Project is running at [http://0.0.0.0:8080/](http://0.0.0.0:8080/)
[react] ℹ ｢wds｣: webpack output is served from /
== trimmed for brevity ===
[react] [./node_modules/webpack/buildin/module.js] (webpack)/buildin/module.js 497 bytes {0} [built]
[react] ℹ ｢wdm｣: Compiled successfully.

3. Review output

From the output we can see a few of things have happened:

• Docker has successfully built the container image

• 1 service was created in the cluster

• 1 deployment was created in the cluster

Lastly, you will see the logs report the deployment has stabilized and then starts logging the stdout from the container. React is running the container and watching for changes, as changes are detected they will be reported here.

4. Find project URL

So let’s navigate to our sample app and check it out. Near the beginning of the react output you will see that react reports where the application is running. This should match up with the port defined in Dockerfile and manifest files.

Project is running at [<http://0.0.0.0:8080/>](<http://0.0.0.0:8080/>)

5. Open Sample App

Let’s navigate this to this address in your local web browser. If all went well you should see Hello world! in large text in the middle of the window.

6. Modify HelloWorld.js

Now let’s modify some of the configuration files and watch how Skaffold handles it. Using a text editor or IDE open the file: app/src/components/HelloWorld.js

Let’s make it a bit more interesting and have it say Hello From Skaffold!

import React from ‘react’;
import ‘../styles/HelloWorld.css’;
export const HelloWorld = () => (
<div>
<h1>Hello From Skaffold!</h1>
</div>
);

7. Watch file sync in action

Remember the entire project folder is being watched by Skaffold. So the second you save the changes to this file, Skaffold will kick off the update process. Looking back at your web browser you should see that the changes should already be displayed. And from the log output we see that Skaffold reports 1 file has been synced with the running container.

Syncing 1 files for react-reload:5212d19febc7b8a0e5b623473c502a339b7222034e64024a3c3c45e3ab0e3281
Watching for changes…
[react] ℹ ｢wdm｣: Hash: f8313748da8d23ab6dc6
[react] Version: webpack 4.46.0
[react] Time: 120ms
[react] Built at: 01/14/2021 9:05:52 PM
[react] Asset Size Chunks Chunk Names
[react] fa444694c556f0f903da.hot-update.json 46 bytes [emitted] [immutable] [hmr]
[react] index.html 295 bytes [emitted]
[react] main.fa444694c556f0f903da.hot-update.js 1.39 KiB main [emitted] [immutable] [hmr] main
[react] main.js 1.36 MiB main [emitted] main
[react] Entrypoint main = main.js main.fa444694c556f0f903da.hot-update.js
[react] [./src/components/HelloWorld.js] 210 bytes {main} [built]
[react] + 52 hidden modules
[react] Child html-webpack-plugin for “index.html”:
[react] 1 asset
[react] Entrypoint undefined = index.html
[react] 4 modules
[react] ℹ ｢wdm｣: Compiled successfully.

8. Update dependencies

Now let’s make a change that will cause a bit of a bigger splash; let’s modify the package.json file in the app folder. We can pretend that we need an older version of React. Downgrade the react dependency from 16.13.1 to 16.13.0

“dependencies”: {
    “react”: “¹⁶.13.0”,
    “react-dom”: “¹⁶.13.1”
},

9. Watch automatic redeploy

Unlike the change earlier which did a hot-swap the file javascript file, Skaffold detected the change in dependencies and kicked off a new build for the container image. But we can see docker was able to rebuild the first 4 layers from the cache before copying in the modified package.json file and building the new layer with npm install. Then Docker hands off this newly made image back off to Skaffold where it updates the running deployment.

Generating tags…
- react-reload -> react-reload:v1.17.1–57-g95092addf-dirty
Checking cache…
- react-reload: Not found. Building
Found [docker-desktop] context, using local docker daemon.
Building [react-reload]…
Sending build context to Docker daemon 11.26kB
Step 1/7 : FROM node:12.16.0-alpine3.10
— -> 48ae9395145d
==trimmed for brevity===
— -> ff4ad02e16bb
Step 7/7 : COPY . .
— -> ae2a7b322a8f
Successfully built ae2a7b322a8f
==trimmed for brevity===

10. Conclusion

You should now see your local application running with the updated package dependencies. This alone is a huge time saver, thinking about how often you need to rebuild an image due to a small file change, not to mention redeploying after that. But now let’s look at how we can make this inner-dev-loop even faster; with the addition of Telepresence, we can project our local dev environment into an external cluster.

Skaffold dev and Telepresence for fast, realistic testing in dev

Why introduce Telepresence when we already have a perfectly working development environment? Each of these tools saves time for a very specific piece of the inner-dev-loop, this is what makes them a dynamic duo. And to reiterate the inner-dev-loop is to write code, deploy code, test code and repeat.

Skaffold excels at deploying code to your local development environment, saving you time from running all the operation bits like docker build and kubectl apply. Telepresence on other hand saves time testing code. By intercepting traffic from remote clusters, it’s almost like testing your local services directly in staging or production clusters. Together they create an inner-dev-loop that provides nearly instantaneous feedback on how your code will perform in the production environment. Building on top of example Skaffold is running in Docker.

For today’s example, we’ll be using Telepresence as part of Ambassador Cloud as Ambassador gives us some additional features we’ll want to take advantage of in later parts of this tutorial and future tutorials. Let’s start by projecting this local service into a remote cluster. While it is possible to complete this with another cluster hosted on your local machine, I highly recommend a cluster that is hosted with a Cloud provider. To use Ambassador Cloud, you’ll be asked to log in with GitHub to create a cloud account. Then, you’ll see the instructions to install and configure Telepresence.

Prerequisites

1. The react-reload demo running from the previous section

2. A Kubernetes cluster hosted with a Cloud provider

3. The Telepresence binary, Install instructions can be found once you login here: https://app.getambassador.io/cloud/preview/

Check requirements

1. Check the demo is still running in your local browser

2. Create a new terminal window with kubectl connected with your cloud-hosted cluster. I’m setting this new terminal to use a new kubeconfig file, but you can also switch context.

   $ export KUBECONFIG=/home/cloud-cluster.yml
   $ kubectl get nodes
   NAME STATUS ROLES AGE VERSION
   cloud-cluster Ready master 10d v1.17.3+k3s1

   1. Verify telepresence is installed and working

   $ telepresence version
   Client v2.0.0 (api v3)

Example Steps

1. Upload container image to Docker hub

For Telepresence to intercept traffic we need our sample application to also be running in our Cloud cluster. To do this we need to first upload the container image created by Skaffold to Dockerhub.

Let’s find the image id with docker image ls

$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
peteroneilljr/react-reload e2a7b322a8f36cf26ff3c12aaa156d8176fb42f26a0ceba3e241b6350c6c01c ae2a7b322a8f 2 hours ago 174MB
peteroneilljr/react-reload v1.17.1–57-g95092addf-dirty ae2a7b322a8f 2 hours ago 174MB
peteroneilljr/react-reload 5212d19febc7b8a0e5b623473c502a339b7222034e64024a3c3c45e3ab0e3281 5212d19febc7 3 hours ago 174MB
=== trimmed for brevity ===

We see at least a few different react reload images. The first image created is the one we want to upload, the newer images were created by Skaffold when we made modifications. So let’s give the base image a tag of “latest” and push it to Dockerhub

$ docker image tag 5212d19febc7 peteroneilljr/react-reload:latest
$ docker push peteroneilljr/react-reload
Using default tag: latest
The push refers to repository [docker.io/peteroneilljr/react-reload]
ef61bb2587ca: Pushed
34561a8deb68: Pushed
e2718bad0837: Pushed
3687ae930f2e: Pushed
5947f76477a5: Mounted from library/node
8b86a15dc982: Mounted from library/node
66d69d514191: Mounted from library/node
531743b7098c: Mounted from library/node
latest: digest: sha256:91676ba8810f01b39416f5adaf7cb2c47e82b6a42e58e0735ab6f4f47fb78902 size: 1992

2. Run the sample app in the Cloud cluster

Now that our image is hosted in a public repository we can recreate our sample application in the cloud cluster. Remember the kubernetes manifest? We had one deployment and one service. Let’s create these now.

$ kubectl create deployment node — image=peteroneilljr/react-reload — port=8080
deployment.apps/node created
$ kubectl expose deployment node — type=LoadBalancer — port=8080 — target-port=8080
service/node exposed

3. Check that the sample app is running

Next, let’s find the IP address assigned to our new application.

$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
node LoadBalancer 10.43.95.91 35.223.x.x 8080:32708/TCP 2m52s

And let’s navigate to this address and port in our browser. http://35.223.x.x:8080

4. Intercept traffic

Time to start Telepresence! Let’s start with a telepresence list to verify everything is working as expected.

$ telepresence list
Launching Telepresence Daemon v2.0.0 (api v3)
Need root privileges to run “/usr/local/bin/telepresence daemon-foreground ‘’ ‘’”
Password:
Connecting to traffic manager…
Connected to context default (https://34.123.x.x)
node : ready to intercept (traffic-agent not yet installed)

We should see the node deployment is ready to intercept! So now we are going to intercept the node deployment in the public cluster with the service running on our local port 8080.

$ telepresence intercept node — port 8080
Using deployment node
intercepted
State : ACTIVE
Destination : 127.0.0.1:8080
Intercepting: all connections

Now if you navigate back to your web browser and refresh page, you should see your local deployment being displayed on the public IP address from your Cloud cluster.

5. Testing our inner-dev-loop

With Skaffold and Telepresence working together you’re able to see the changes to our local cluster get populated on the public IP address in real time. Let’s test this out, this time we are going to modify the HelloWorld.css file in the app/src/styles/ folder. Update the h1 color to be ff0000

h1 {
  color: #ff0000;
  text-align: center;
  margin-top: 40vh;
  font-size: 120pt;
}

Once you save this file we should see the heading of the page in your browser pointed at your local cluster change to red nearly instantly, and very shortly after you should notice your public ingress change as well.

6. Sharing our intercept with teammates using Ambassador Cloud

Because WebPack has host checking enabled by default we will need to disable it for this next part.

Open app/webpack.config.js

Add disableHostChecking: true to devServer

devServer:{
  host: ‘0.0.0.0’,
  disableHostCheck: true
},

Save this file and Skaffold will rebuild your local image.

Tag and push this new image to Dockerhub

$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
peteroneilljr/react-reload 82d036eee5887d5660e0a787435f01fddec66c0617c8dc364f45421941aa4f9e 82d036eee588 19 hours ago 173MB
peteroneilljr/react-reload latest 82d036eee588 19 hours ago 173MB
$ docker tag 82d036eee588 peteroneilljr/react-reload
$ docker push peteroneilljr/react-reload

Delete the current pod so that it can respawn with the new configuration.

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
node-86dfc695f5–22psv 1/1 Running 0 13s
$ kubectl delete pod node-86dfc695f5–22psv

Now let’s leave the current intercept

$ telepresence leave node

Login to telepresence, this will take you to GitHub to authenticate.

$ telepresence login
Launching browser authentication flow…
Login successful.

Create a new intercept. This time Telepresence will ask a few questions. Follow the prompts shown below.

$ telepresence intercept node — port 8080
Confirm the ingress to use for preview URL access
Ingress service.namespace [ambassador.ambassador] ? node.default
Port [80] ? 80
Use TLS y/n [n] ? n
Using deployment node
intercepted
State : ACTIVE
Destination : 127.0.0.1:8080
Intercepting: HTTP requests that match all of:
header(“x-telepresence-intercept-id”) ~= regexp(“6e5b52d3–676a-421d-8223–397fb1146162:node”)
Preview URL : [https://nostalgic-colden-2717.preview.edgestack.me](https://nostalgic-colden-2717.preview.edgestack.me)

Now our intercept can be seen and shared with the preview URL.

7. Cleaning up

Just like that we’ve reached the end of our demo, so let’s clean up all the resources we’ve used.

First Telepresence, either leave the intercept if you plan to use it more or uninstall everything. With either approach you’ll see the display quickly return back to the original prompt and color to show the intercept is not longer applied.

$ telepresence leave node
$ telepresence uninstall — everything

Next Skaffold, you can stop skaffold dev by pressing ctr+c within the terminal window.

Lastly, the Cloud cluster. If you’re going to keep using the cluster remove the resources we created, otherwise feel free to delete the whole cluster.

$ kubectl delete svc node

service “node” deleted

$ kubectl delete deploy node

deployment.apps “node” deleted

Conclusion

Now as we can see from this example there are a lot of moving pieces when it comes to writing code for a Cloud Native application. Luckily there are tools like Skaffold that save time managing container images and Kubernetes deployments, and tools like Telepresence & Ambassador to save time debugging by creating a better testing environment.

If you’re considering using Skaffold or just exploring possible tools to make your development workflow better, I hope this article helped out. Drop a comment and let me know what you liked, what you didn’t like, or what you want to hear more about. You can also Tweet at me to let me know what you think @peteroneilljr

Here are some additional resources that may help you get started:

• Skaffold documentation

• Watch the demo video of Ambassador Cloud and Telepresence

• Learn more about Ambassador Cloud and Telepresence

• Join our Slack channel