Tailscale Kubernetes Operator

Apologizes to anyone looking for more #Pidgin content. I know it's been awhile and I'll get something out soon I promise, but there are quite a few Pidgin related things in this post too...

About a month ago I setup a Kubernetes cluster using Talos to handle my container load at home.

I have containers for all sorts of things including the graphs you all may have seen on my stream as well as an Ergo instance for developing our new IRCv3 protocol plugin.

I'll be adding to this cluster in the future too as we continue development on Xeme which is our new XMPP library as well as Myna which is our new Matrix library.

Having these servers running locally makes it easier to test interesting configurations, but more importantly, lets me use them on stream without leaking my IP address or any passwords.

Anyways, I'm not always at home when I want to work on something and I wanted to find a way to access services remotely. Well tonight I realized I should be able to do this with Tailscale.

I'm already using Tailscale to make sure we can access all of our build agents which are spread across a few sites, so being able to expose this to the other Pidgin developers who have access to that Tailscale network is just icing on the cake.

So I decided to give the Tailscale Kubernetes Operator a go. They have full documentation it here. As is pretty typical right now their installation instructions are based on using Helm.

I prefer to use Kustomize over Helm. There's plenty of reasons, but there's no reason to get into that now ;)

Thankfully, Tailscale provides a static manifest for deploying the operator. This is very easy to import into our kustomize and get running directly quite quickly.

But of course, it's not just that easy. Their instructions have you modifying that manifest and then applying that. This isn't a big deal, but it makes upgrading harder as you have to keep track of what you edited. Sure you could use version control (and you should have your manifest in version control) but kustomize makes this very easy.

First we needed to download operator.yaml which is the static manifest that Tailscale provides. We're going to use this file as is without modification which means we can just replace it if/when they update it.

Next we need to create our kustomization.yaml which is used to drive everything. My commented kustomization.yaml is below.

---
# Here we reference the unmodified operator.yaml we downloaded from
# Tailscale.
resources:
  - operator.yaml
patches:
  # Tailscale needs network admin and some other permissions, so
  # we use the following patch that will add the appropriate
  * annotations to the namespace.
  - patch: |-
      apiVersion: v1
      kind: Namespace
      metadata:
        name: tailscale
        labels:
          pod-security.kubernetes.io/enforce: privileged
          pod-security.kubernetes.io/audit: privileged
          pod-security.kubernetes.io/warn: privileged
secretGenerator:
  # As I mentioned earlier, operator.yaml defines a secret for the
  # tailscale oauth secrets which the documentation wants you to
  # modify manually. We can instead use a secret generator to read
  # those values from an `env` file and merge them into the
  # existing secret.
  - name: operator-oauth
    namespace: tailscale
    behavior: merge
    envs:
      - secrets/env

With this all now setup, we can run kubectl kustomize and verify our output. A condensed output with just the namespace and secret are below.

apiVersion: v1
kind: Namespace
metadata:
  labels:
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/warn: privileged
  name: tailscale
---
apiVersion: v1
data:
  client_id: aGFjayB0aGUgcGxhbmV0
  client_secret: aGFja2VycyBvZiB0aGUgd29ybGQgdW5pdGU=
kind: Secret
metadata:
  name: operator-oauth
  namespace: tailscale
type: Opaque
---

Now that we've confirmed everything looks good, we can get ready to apply it to our cluster. Normally I'd recommend doing a dry run first, but since we're applying a namespace as well most of the tests will fail because that namespace doesn't exist.

Anyways, we can apply this kustomization with the following command:

kubectl apply -k .

If everything is successful you'll see something like the following:

namespace/tailscale configured
customresourcedefinition.apiextensions.k8s.io/connectors.tailscale.com configured
customresourcedefinition.apiextensions.k8s.io/dnsconfigs.tailscale.com configured
customresourcedefinition.apiextensions.k8s.io/proxyclasses.tailscale.com configured
serviceaccount/operator configured
serviceaccount/proxies configured
role.rbac.authorization.k8s.io/operator configured
role.rbac.authorization.k8s.io/proxies configured
clusterrole.rbac.authorization.k8s.io/tailscale-operator configured
rolebinding.rbac.authorization.k8s.io/operator configured
rolebinding.rbac.authorization.k8s.io/proxies configured
clusterrolebinding.rbac.authorization.k8s.io/tailscale-operator configured
secret/operator-oauth configured
deployment.apps/operator configured
ingressclass.networking.k8s.io/tailscale configured

You can verify the operator is up and running the following command:

kubectl -n tailscale get deployments

You should see something like the following:

NAME       READY   UP-TO-DATE   AVAILABLE   AGE
operator   1/1     1            1           72m

If that all looks good you can go ahead and verify that it shows up in the Machines section in Tailscale. You should see something like the following:

Now we're ready to expose a service via Tailscale!

As I mentioned earlier, I'm running an Ergo instance for local IRC development. I have an Service of type LoadBalancer to expose it to my LAN via MetalLB.

The Tailscale operator has multiple ways to integrate with your Kubernetes cluster, but we're just going to expose existing services to the tailnet and doing that is extremely simple.

To do so, you just need to add the tailscale.com/expose: "true" annotation to the service. Below is my updated Service for Ergo:

---
apiVersion: v1
kind: Service
metadata:
  name: ergo
  annotations:
    tailscale.com/expose: "true"
  labels:
    app: ergo
spec:
  ports:
    - port: 6667
      protocol: TCP
      name: irc
    - port: 6697
      protocol: TCP
      name: ircs
  selector:
    app: ergo
  type: LoadBalancer

Now that that's done, I can apply the Kustomization for Ergo:

kubectl apply -k .

The operator will automatically add it to the tailnet with a machine name of <namespace>-<service-name>. This ergo instance is running in the reaperworld namespace so it's machine name, and therefore DNS name is reaperworld-ergo which we can now see in the Tailscale machines page.

And that's it, that service is now available on the tailnet accessible to anyone else on the tailnet!

I hope you all found this interesting. I decided to write this as I had to learn about behavior: merge property of the secretGenerator and figured others might find this useful as well. Plus it gave me an excuse to show the power of Kustomize!