Originally published in Admiralty's blog.

If you operate multiple Kubernetes clusters, and need a pod in a cluster to call the Kubernetes API server of another cluster, we know your pain. Up until now, you either had to rely on heterogeneous identity providers outside Kubernetes, or repurpose remote service accounts the hard way.

A Kubernetes cluster typically authenticates API clients using two or more authentication modules. One module authenticates Kubernetes service accounts, defined in-cluster; the other modules authenticate users, defined outside the cluster.

User authentication depends on your Kubernetes distribution: it can simply be based on static files containing lists of passwords, tokens and/or client certificates; it often uses Open ID Connect; it can also use other methods (LDAP, SAML, Kerberos, etc.) via custom webhooks and proxies. Even with the common Open ID Connect module, the backing identity provider depends on your distribution: Google Cloud IAM, Azure Active Directory, among others.

Therefore, in a hybrid and/or multicloud architecture, Kubernetes user authentication can be heterogeneous. A pod in a cluster trying to call the API servers of other clusters as a "user" may have to login to several identity providers. Even if you use a single identity provider across clusters, you're still responsible for mounting secrets inside pods—client-go's authentication automation is designed for out-of-cluster, human-facing tools. To keep things simple, we need to agree on a common, automated multicluster identity provider for pods.

Kubernetes may well be the identity provider we're looking for. It already provides identities to pods: each pod authenticates with its local Kubernetes API server using a service account in its namespace. Kubernetes even automates token generation and pod configuration:

1. The service account controller ensures that at least a "default" service account exists within each namespace, though more can be created.
2. The token controller generates a token for each service account and puts it in a secret in the same namespace.
3. The service account admission controller automounts a service account secret in each pod, either for the "default" service account or the one specified in the pod spec.

Combined with role-based access control (RBAC), service accounts are a powerful identity and access management (IAM) solution for the Kubernetes API. However, although a service account's token can be used outside its namespace and cluster, the automated token generation and volume mounts only work in the service account’s namespace; it is your reponsibility to get an up-to-date token and configure your client accordingly.

A few multicluster tools have repurposed kubeconfig files to store remote service account tokens, because they can also store remote API server URLs and root certificates, and they are supported by client-go. That's what we used at first when we open-sourced multicluster-controller, but that was a bit of a hack. Kubeconfig files are designed to be used out-of-cluster by Kubernetes clients like kubectl to authenticate actual users, but nothing prevents us from mounting them inside pods to authenticate remote service accounts. However, that just solves one part of the problem and kubeconfig files must still be generated and mounted inside pods.

Today, we're open-sourcing multicluster-service-account, leveraging service accounts as multicluster identities, with all the necessary automation:

1. a ServiceAccountImport custom resource definition (CRD) and controller to import remote service accounts (and their secrets);
2. a dynamic admission webhook to automount service account import secrets inside annotated pods, the same way regular service accounts are automounted inside pods;
3. a Go library of helper methods to generate client-go configurations from service account imports (as well as generic methods to fall back to kubeconfig contexts and regular service accounts).

Check out the README for more details, including setup instructions. Contributions, feature requests and bug reports are always welcome.

Note: The problem space of multicluster-service-account is the authentication of Kubernetes clients only. For service-to-service multicluster authentication, we recommend Istio multicluster.