Transient Tech: JYSK’s Journey to On-Demand Development with Ephemeral Workspaces

· Benefits of Ephemeral Environments

· How does JYSK implement this?

· Getting Started

· Coder deployment

· Coder CLI installation and setup

· Creating a Basic Workspace

· Tips & Tricks

· Automatic Port Forwarding

· Using SCP to transfer files

· Predefined Templates

· SSH keypairs

· Remove all workspaces, templates at once

· Conclusion

In recent years, ephemeral dev environments have become increasingly popular among developer/ops engineers. They provide a convenient and efficient way to create temporary development environments. Unlike traditional development environments that require complex setup and configuration, ephemeral environments can be created on-demand, used for a specific task, and discarded once the task is complete. This approach has several advantages, including reducing the overhead of managing complex development environments, improving reproducibility, and increasing flexibility.

In JYSK Ops we have started to adopt such environments, what we call as “JYSK Workspaces”, and in this blog post / deep-dive , we will explore the concept of ephemeral coding environments in more detail, discussing their benefits, use cases, and how they can be implemented… or at least try to.

Benefits of Ephemeral Environments

Ephemeral development environments have several benefits and not just for development teams:

1. Increased Flexibility: Environments can be quickly spun up and down, providing teams with greater flexibility to adapt to changing requirements and workloads. This is particularly useful for DevOps teams that need to support multiple projects simultaneously.
2. Improved Reproducibility: Ephemeral environments can be created from the same configuration files and scripts, ensuring that every developer is working with an identical environment. This can help to improve the reproducibility of builds and deployments, reducing the likelihood of errors or inconsistencies.
3. Reduced Overhead: Ephemeral environments eliminate the need for maintaining and updating long-lived development environments, reducing the overhead and costs associated with managing complex development infrastructure.
4. Increased Security: These environments can be designed to be more secure than traditional development environments, as they can be destroyed after use, reducing the risk of sensitive data being left on the system. In addition to, these environments reduce their blast radius.
5. Simplified Testing: Ephemeral environments can be used to quickly spin up test environments that are identical to production, allowing for more accurate testing and reducing the risk of bugs or errors being introduced during deployment.

Overall, ephemeral development environments can help DevOps teams to improve their efficiency, agility, and security, while also reducing the complexity and overhead of managing development infrastructure.

How does JYSK implement this?

At JYSK Ops, we’ve begun embracing what we term “JYSK Workspaces”. In this article/deep-dive, we aim to delve deeper into the idea of ephemeral coding environments, touching on their advantages, applications, and methods of implementation… or we’ll give it our best shot.

The requirements for coding ing JYSK ops are currently quite flexible. We have basic style guides but do not mandate the language, allowing us a wide range of choices. In our development department, there of course are specifics, and these are created by developers; for developers. While this freedom has its advantages, it also has downsides. Implementing guidelines in our ops-development environment assures the business that essential parameters, such as using our developer security platform and adhering to coding standards, are being followed.

I recently participated in a CNCF workshop where they showcased Sysdig’s Falco Container Runtime Monitor. see what the hell is going on inside my container. They utilized GitHub’s Codespaces for this, and I was instantly hooked. The convenience of accessing a project, making edits, and committing changes all through a browser from any location truly sparked my interest.

Today, our operations teams remain dedicated across various sectors, from Store IT operations that support specific applications/environments to our network infrastructure which is embarking on a robust automation journey. While these departments aren’t centered on development, they can seamlessly integrate with the established DevOps principles within our business. This fosters a tighter connection between operations and development, promoting a more cohesive workflow.

Why not establish a secure environment where our operations teams can write code, aligned with the existing business standards? A workspace that’s both secure and efficient, equipped with our built-in proxy configuration and locally configured repositories. Where one can either initiate a new project or fetch an existing one. Upon committing, pipelines handle everything else, leading to teams or individuals who can swiftly produce results and access a comprehensive suite of scanning, building, and deploying pipelines without further requests. If this is already a staple in most of our development sectors, why not extend it to operations — or even further?!

Getting Started

This assume we already have a working Kubernetes cluster. Currently i am using a dev environment we have internally for this, but something like k3s/k0s etc will also work fine.

If users are to provision their own persistent volumes, some shared storage is needed. We are using an NFS Provisioner.

Coder deployment

A basic helm install on our Kubernetes setup is all it takes. We are adding some extra variables on deployment for integration with our OIDC — Keycloak.

Coder have an excellent guide on getting this up and running here so I will not repeat. For this demo i am using coder v2 — helm repo add coder-v2 https://helm.coder.com/v2

The only modifications are:

- name: CODER_ACCESS_URL
  value: 'https://workspaces-poc.jysk.local'
- name: CODER_WILDCARD_ACCESS_URL
  value: '*.workspaces-poc.jysk.local'
- name: CODER_OIDC_SIGN_IN_TEXT
  value: 'Sign in with Keycloak'
- name: CODER_OIDC_ICON_URL
  value: 'https://upload.wikimedia.org/wikipedia/commons/2/29/Keycloak_Logo.png'
- name: CODER_OIDC_ISSUER_URL
  value: 'https://<oidc domain>/auth/realms/dev'
- name: CODER_OIDC_EMAIL_DOMAIN
  value: 'jysk.com'
- name: CODER_OIDC_CLIENT_ID
  value: 'workspaces-poc'
- name: CODER_OIDC_CLIENT_SECRET
  valueFrom:
    secretKeyRef:
      name: 'oidc-secret'
      key: 'secret'
- name: CODER_OIDC_IGNORE_EMAIL_VERIFIED`
  value: 'true'

CODER_OIDC_CLIENT_SECRET comes from the secret oidc-secret

k create secret generic oidc-secret -n coder --from-literal=secret=<your secret here>

If one were to utilize a corporate proxy with TLS interception we need to give our coder installation the needed CAs, the following is added to our values.yaml file

certs:
  secrets:
    - name: 'certs'
      key: 'jysk_ca_bundle.cer'

We will also need a secret for the PostgresSQL DB, and can be set with CODER_PG_CONNECTION_URL. If this is not set, then coder will use an internal Postgres DB however this will of course be temporary and removed when the container dies.

k create secret generic coder-db-url -n coder \
   --from-literal=url="postgres://coder:coder@coder-db-postgresql.coder.svc.cluster.local:5432/coder?sslmode=disable"

Other than that it’s a basic update/install

➜ coder git:(main) ✗ helm upgrade -i coder coder-v2/coder --namespace coder -f values.yamlRelease "coder" has been upgraded. Happy Helming!

NAME: coder
LAST DEPLOYED: Mon Aug 14 09:09:17 2023
NAMESPACE: coder
STATUS: deployed
REVISION: 23
TEST SUITE: None
NOTES:
Enjoy Coder! Please create an issue at [https://github.com/coder/coder](https://github.com/coder/coder) if you run

Coder CLI installation and setup

In addition to the environment, we also need a client cli. This give the ability to easily interact with the new coder installation. Consult the coder v2 docs for more info, but it can easily be installed with:

curl -fsSL https://coder.com/install.sh | sh

When installed, we need to connect our cli to our new instance, as we have defined the ingress as workspaces-poc.jysk.local and our auth method OIDC then we can use:

➜ coder git:(main) ✗ coder login https://workspaces-poc.jysk.local/
Your browser has been opened to visit:

https://workspaces-poc.jysk.local/cli-auth

> Paste your token here: <token>
> Welcome to Coder, rgo! You're authenticated.

From here on our we mostly use the cli to configure our workspaces. Lets make a basic workspace so we can edit this tech blog.

Creating a Basic Workspace

The coder docs actually have a really nice example of a basic setup. However, we require a little more configuration for an enterprise setup. Namely, we want proxy support and our own CA’s installed. In addition, there are also some environment settings and some basic lint packages installed.

Before we proceed — lets make a custom container image to deploy for our ubuntu workspace. Again, we are running a an air-gapped setup, thus we pull down through a proxy, install our certificates and forward proxy settings for apt, in addition we run the code-server setup in our container. This is mainly to improve deployment speed.

FROM <domain>/dockerhub/codercom/enterprise-base:ubuntu

USER root

RUN cd /tmp && curl -O http://<domain>/JYSK_certificates.zip
RUN mkdir -p /usr/local/share/ca-certificates/jysk && \
    unzip -j /tmp/JYSK_certificates.zip *.cer -d /usr/local/share/ca-certificates/jysk
RUN cd /usr/local/share/ca-certificates/jysk && for file in *.cer; do mv "$file" "${file%.cer}.crt"; done
RUN /usr/sbin/update-ca-certificates

RUN echo 'Acquire::http::proxy "http://<proxy address>:8080";' > /etc/apt/apt.conf.d/proxy.conf
RUN echo 'Acquire::https::proxy "http://<proxy address>:8080";' >> /etc/apt/apt.conf.d/proxy.conf

# install the latest version
RUN curl -fsSL https://code-server.dev/install.sh -v | sh
USER coder

This is built, and pushed to our internal container registry:

docker build -t <domain>/devops/jysk-enterprise-base:ubuntu
docker push

After addressing this, we can establish our initial template, which we’ll name jysk-ubuntu-workspace. By executing coder templates init, a list of templates will be presented, choose the Kubernetes template, and it will produce a Terraform file.

We’ve updated the Terraform manifest and integrated our container image (previously created) within the "kubernetes_deployment" "main" block's container definition. Finally, we've incorporated startup configurations that install a sample VSCode extension like code-spell-checker, set up basic Git defaults such as name and email, and naturally, incorporated our essential proxy settings and CA certificates for the node.

TIP! If using a forward proxy, take note of the $KUBERNETES_PORT_443_TCP_ADDR added to the NO_PROXY environment variable. This is needed so that coder doesn't try and contact the mothership (the coder container) through our proxy!

resource "coder_agent" "main" {
  ...

  startup_script = <<-EOT
    set -e
    /usr/bin/code-server --install-extension streetsidesoftware.code-spell-checker
    /usr/bin/code-server --auth none --port 13337 >/tmp/code-server.log 2>&1 &
  EOT

  env = {
    GIT_AUTHOR_NAME = "${data.coder_workspace.me.owner}"
    GIT_COMMITTER_NAME = "${data.coder_workspace.me.owner}"
    GIT_AUTHOR_EMAIL = "${data.coder_workspace.me.owner_email}"
    GIT_COMMITTER_EMAIL = "${data.coder_workspace.me.owner_email}"

    HTTP_PROXY = "http://<proxy address>:8080"
    HTTPS_PROXY = "http://<proxy address>:8080"
    NO_PROXY = "jysk.com,jysk.local,$KUBERNETES_PORT_443_TCP_ADDR"
    NODE_EXTRA_CA_CERTS = "/etc/ssl/certs/ca-certificates.crt"
  }

  ...

Finally, we can now create our template.

# Create the template
coder templates create jysk-ubuntu

# Check it has been applied
code template list

Once our template is created, we can then launch our workspace — this can be done via the web UI or command line. Any of the users that have access to the system can now login, and create their own personal workspace. In the CLI we can create a basic workspace:

# Create a workspace from our jysk-ubuntu template
coder create --template="jysk-ubuntu" jysk-tech

# Make sure we are up and running
coder list -a

# We can also check this in our k8s api
kubectl get pod -n coder

A full demo of the setup:

untitled

We can also create a workspace via the UI

Tips & Tricks

Automatic Port Forwarding

Coder automatically publishes any ports opened within your VSCode. So for example, when creating this blog post i am working within my workspace, and have hugo serve running in my intergrated terminal. This then provides me with a URL, so i can see live updates as i type.

One small caveat with this is that you need to have a wildcard tls certificate as coder uses subdomains to determine the port that needs to be forward.

hugo serve -b https://443--main--jysk-tech--rgo.workspaces-poc.jysk.local/ -p 443

Of-course the URL will differ on your installation, but the basic idea is that i can write and view at the same time.

VSCode and Hugo

Using SCP to transfer files

Whilst writing this blog, in coder i needed to be able to transfer some of the demo files and screenshots. Coder gives an SSH option that’s very simple to setup.

Transfering files can also utilize this method:

# Setup the .ssh/config file for your instance 
coder config-ssh

# transfer the file
scp /Users/rgo/Screenshots/ssh.png coder.jysk-tech:website/content/posts/ephemeral-environments/

In addition to this you can just drag & drop files into the UIs folder structure.

Predefined Templates

Some of the great features of coder is pre-defined workspaces. In JYSK we have created templates for Golang, Powershell, basic Ubuntu and even a VNC Remote Desktop. Applied with some default business configuration it makes spinning up a new workspace easy. This cuts time in onboarding consultants and third party users.

SSH keypairs

By default, coder creates an ssh key for every user on the system. Therefore, on the users profile you can find the public part which can be used in Github. This makes pulling down repositories a breeze!

Remove all workspaces, templates at once

The below commands will remove all workspaces and templates — including any user data! Use with care.

Remove all workspaces:

for i in `coder list -a | tail -n +2`; do coder delete $i -y
done

Remove all templates:

for i in `coder template list | tail -n +2`; do coder template delete $i -y
done

Conclusion

Ephemeral dev environments, like the “JYSK Workspaces” we’ve introduced at JYSK Ops, represent a paradigm shift in how developer/ops engineers approach their work. These environments, which can be spun up on-demand and discarded post-task, usher in enhanced efficiency, reproducibility, and adaptability. As the tech landscape continues to evolve, embracing such innovations ensures that we remain agile, reducing unnecessary overheads and fostering an environment conducive to streamlined operations and optimal productivity.

Although this has been a proof of concept within the company, we have been using it on/off for some months. Interest has grown and eventually this will be set into a productive setting where we can make better use.

For now though, it’s a great playground to test coder on!

Happy coding!