Kubernetes is a complex system that relies on several components being configured correctly to have a working cluster.
Some solutions already exist to automate clusters' creation, upgrade, deletion, like for instance ClusterAPI. Such solutions require the existance of a management Kubernetes cluster from where tens of other Kubernetes clusters are then programmatically created and managed.
Managing the life-cycle of Kubernetes clusters is the first step. After a Kubernetes cluster is created, an handful number of features needs to be installed (CNI for instance). Kubernetes add-ons are used to extend the functionality of Kubernetes clusters.
So after cluster creation, the next problem that needs to be addressed is how to programmatically manage Kubernetes add-ons in each managed cluster from the management cluster. Luckily, even in this space, there are already existing solutions. For instance, Sveltos is an open source project to programmatically deploy Kubernetes add-ons in tens of Kubernetes clusters.
Summarizing, using a Kubernetes cluster to manage other clusters (creation/upgrade/deletion and add-ons deployment) provides several benefits, including:
- Centralized management: A cluster management cluster allows administrators to manage multiple clusters from a single place, making it easier to maintain consistency and reduce the risk of configuration issues.
- Scalability: A cluster management cluster can help organizations scale their infrastructure by making it easier to create, deploy, and manage multiple clusters.
- Better security: A cluster management cluster can be configured with security-related addons, such as network policies and secrets management, to help ensure that all managed clusters are secure.
- Increased automation: A cluster management cluster can be integrated with a continuous integration/continuous deployment (CI/CD) pipeline, making it easier to automate the deployment of new clusters and addons, and reducing the time and effort involved in managing the infrastructure.
Problem gets more complex when different organizations end up sharing the managed Kubernetes clusters. This scenario is commonly referred to as multi-tenancy.
Common forms of multi-tenancy are:
- share a cluster between multiple teams within an organization, each of whom may operate one or more workloads;
- one (or more) cluster(s) fully reserved for an organization.
In this scenario we need to introduce two roles:
- platform admin: The role of a platform administrator is to manage and maintain the infrastructure of the managed Kubernetes clusters. This includes tasks such as setting up the cluster, managing nodes, installing and updating software components, and ensuring the overall health and security of the cluster. The platform administrator is also responsible for onboarding tenant admins and granting permissions to those;
- tenant admin: Has admin access to the clusters/namespaces assigned to them by the platform admin and manages tenant applications in those clusters/namespaces.
After cluster creation/upgrade/deletion and the addons deployment, the next problem to solve is how to allow platform admin to onboard tenants and programmatically grant them permissions from the management cluster.
Sveltos addons management solution
Before we deep dive into granting permissions to tenants, lets briefly summarize Sveltos solution to manage Kubernetes addons.
Sveltos introduces a single CRD called ClusterProfile for that.
By creating a ClusterProfile instance, we can tell Sveltos:
- where to deploy addons (i.e, in all managed Clusters matching the ClusterSelector field);
- list of addons to deploy (either Helm charts or Kubernetes resource YAMLs contained in referenced ConfigMaps/Secrets).
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
name: deploy-kyverno
spec:
clusterSelector: env=fv
syncMode: Continuous
helmCharts:
- repositoryURL: https://kyverno.github.io/kyverno/
repositoryName: kyverno
chartName: kyverno/kyverno
chartVersion: v2.6.0
releaseName: kyverno-latest
releaseNamespace: kyverno
helmChartAction: Install
policyRefs:
- name: disallow-latest-tag # referenced ConfigMap contains a Kyverno ClusterPolicy
namespace: default
kind: ConfigMap
With Sveltos configuration drift detection we don't have to worry about managed cluster states. Sveltos monitors the state of managed clusters and when it detects a configuration drift for one of the resource deployed because of a ClusterProfile, it re-syncs the cluster state back to the state described in the management cluster.
Sveltos multi-tenancy solution
Let's now go back to the multi-tenancy problem.
We want to:
- allow platform admin to programmatically grant permissions to tenant admins;
- allow tenant admins to manage applications from management cluster, making sure each tenant admin only deploys resources it has permissions to and only in the clusters/namespaces it has permission to.
RoleRequest is the CRD introduced by Sveltos to allow platform admin to grant permissions to various tenant admins.
apiVersion: lib.projectsveltos.io/v1alpha1
kind: RoleRequest
metadata:
name: full-access
spec:
clusterSelector: env=pre-production
admin: eng
roleRefs:
- name: full-access
namespace: default
kind: ConfigMap
where:
- ClusterSelector field is used to select a set of managed clusters where permissions will be granted;
- Admin field selects a tenant admin (later on we will cover how this identify a tenant admin);
- RoleRefs field references one of more ConfigMaps/Secrets. Each of those ConfigMaps/Secrets contains one or more ClusterRole/Role defining the permissions granted to the tenant admin by the platform admin.
Platform admin is the only admin in the management cluster with permissions to create/update/delete RoleRequest instances.
Tenant admin managing applications
Once RoleRequest instances have been created, tenant admins can use Sveltos ClusterProfile CRD to instruct Sveltos on what Kubernetes resources to deploy on which clusters.
Sveltos requires that following label
projectsveltos.io/admin-name: <admin>
is set on a ClusterProfile instances created by a tenant admin.
How to achieve that, is up to platform admin.
For instance, if:
- Kyverno is present in the management cluster;
- tenant admin are represented by Kubernetes ServiceAccount;
following Kyverno ClusterPolicy is all that is needed for label to be automatically added to each instance of a ClusterProfile:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: add-labels
annotations:
policies.kyverno.io/title: Add Labels
policies.kyverno.io/description: >-
Adds projectsveltos.io/admin-name label on each ClusterProfile
created by tenant admin. It assumes each tenant admin is
represented in the management cluster by a ServiceAccount.
spec:
validationFailureAction: enforce
background: false
rules:
- name: add-labels
match:
resources:
kinds:
- ClusterProfile
mutate:
patchStrategicMerge:
metadata:
labels:
projectsveltos.io/admin-name: "{{serviceAccountName}}"
Examples
I have created management cluster on my laptop. It is a Kind cluster.
In the management cluster, I have deployed both ClusterAPI (with Kind as infrastructure provider) and Sveltos.
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
capd-system capd-controller-manager-59b7bdbcf9-zd4xq 1/1 Running 0 30m
capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-controller-manager-68f975d954-9r7lm 1/1 Running 0 30m
capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager-8476fb8676-kpnkd 1/1 Running 0 30m
capi-system capi-controller-manager-69fd995599-67fh5 1/1 Running 0 30m
...
projectsveltos access-manager-569dd6cccd-m676t 2/2 Running 0 59s
projectsveltos classifier-manager-ddc8c756-bmxc9 2/2 Running 0 59s
projectsveltos fm-controller-manager-5964fb8bff-2j2gx 2/2 Running 0 59s
projectsveltos sveltoscluster-manager-75ddcf7cbd-q47hp 2/2 Running 0 58s
I have one ClusterAPI powered cluster:
kubectl get clusters -A
NAMESPACE NAME PHASE AGE VERSION
default sveltos-management-workload Provisioned 10m v1.25.2
and I have one GKE cluster registered with Sveltos
kubectl get sveltoscluster -A
NAMESPACE NAME AGE
gke production 6s
Cluster labels are set as follow:
| Cluster | Label |
|---|---|
| default/sveltos-management-workload | env:development |
| gke/production | env:production |
Fully reserve set of cluster to a tenant
apiVersion: v1
kind: ConfigMap
metadata:
name: full-access
namespace: default
data:
role.yaml: |
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system-admin-full-access
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
---
apiVersion: lib.projectsveltos.io/v1alpha1
kind: RoleRequest
metadata:
name: full-access
spec:
clusterSelector: env=production
admin: system-admin
roleRefs:
- name: full-access
namespace: default
kind: ConfigMap
By referencing ConfigMap default/full-access, above RoleRequest is reserving any cluster matching clusterSelector env=production to admin system-admin.
Using sveltosctl we can verify Sveltos is aware system-admin has full access to managed cluster with label env:production
./bin/sveltosctl show admin-rbac
+-------------------------------+--------------+-----------+------------+-----------+----------------+-------+
| CLUSTER | ADMIN | NAMESPACE | API GROUPS | RESOURCES | RESOURCE NAMES | VERBS |
+-------------------------------+--------------+-----------+------------+-----------+----------------+-------+
| SveltosCluster:gke/production | system-admin | * | * | * | * | * |
+-------------------------------+--------------+-----------+------------+-----------+----------------+-------+
If system-admin tries to deploy addons to such cluster, that operation will go through.
For instance, if system-admin posts following ClusterProfile
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
name: deploy-kyverno
labels:
projectsveltos.io/admin-name: system-admin
spec:
clusterSelector: env=production
syncMode: Continuous
helmCharts:
- repositoryURL: https://kyverno.github.io/kyverno/
repositoryName: kyverno
chartName: kyverno/kyverno
chartVersion: v2.6.0
releaseName: kyverno-latest
releaseNamespace: kyverno
helmChartAction: Install
we can verify Kyverno is deployed by Sveltos on behalf of system-admin:
./bin/sveltosctl show features
+----------------+---------------+-----------+----------------+---------+-------------------------------+------------------+
| CLUSTER | RESOURCE TYPE | NAMESPACE | NAME | VERSION | TIME | CLUSTER PROFILES |
+----------------+---------------+-----------+----------------+---------+-------------------------------+------------------+
| gke/production | helm chart | kyverno | kyverno-latest | 2.6.0 | 2023-02-03 14:06:48 -0800 PST | deploy-kyverno |
+----------------+---------------+-----------+----------------+---------+-------------------------------+------------------+
If system-admin tries to deploy same add-ons on clusters matching clusterSelector env:development that operation will fail.
Share cluster between tenants
Platform admin here is granting following permissions in all clusters matching ClusterSelector env=development
- Admin eng full access to namespace eng and some read permission in namespace shared-service-access;
- Admin hr full access to namespace human-resource.
# ConfigMap contains a Role which gives
# service read access to namespace shared-services
apiVersion: v1
kind: ConfigMap
metadata:
name: shared-service-access
namespace: default
data:
role.yaml: |
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: service-read-role
namespace: shared-services
rules:
- apiGroups: [""]
resources: ["services"]
verbs: ["get,list"]
---
# ConfigMap contains a Role which gives
# full access to namespace eng
apiVersion: v1
kind: ConfigMap
metadata:
name: eng-full-access
namespace: default
data:
role.yaml: |
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: edit-role
namespace: eng
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
---
# RoleRequest gives admin 'eng':
# - full access to namespace eng
# - some read permission in namespace shared-service-access
# in all clusters matching the label
# selector env=development
apiVersion: lib.projectsveltos.io/v1alpha1
kind: RoleRequest
metadata:
name: eng-access
spec:
clusterSelector: env=development
admin: eng
roleRefs:
- name: eng-full-access
namespace: default
kind: ConfigMap
- name: shared-service-access
namespace: default
kind: ConfigMap
# ConfigMap contains a Role which gives
# full access to namespace human-resource
apiVersion: v1
kind: ConfigMap
metadata:
name: hr-full-access
namespace: default
data:
role.yaml: |
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: edit-role
namespace: human-resource
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
---
# RoleRequest gives admin 'hr' access to namespace
# 'human-resource' in all clusters matching the label
# selector env=development
apiVersion: lib.projectsveltos.io/v1alpha1
kind: RoleRequest
metadata:
name: hr-access
spec:
clusterSelector: env=development
admin: hr
roleRefs:
- name: hr-full-access
namespace: default
kind: ConfigMap
As before, we can use sveltosctl to verify granted permission:
./bin/sveltosctl show admin-rbac --namespace=default
+---------------------------------------------+-------+-----------------+------------+-----------+----------------+----------+
| CLUSTER | ADMIN | NAMESPACE | API GROUPS | RESOURCES | RESOURCE NAMES | VERBS |
+---------------------------------------------+-------+-----------------+------------+-----------+----------------+----------+
| Cluster:default/sveltos-management-workload | eng | eng | * | * | * | * |
| Cluster:default/sveltos-management-workload | eng | shared-services | | services | services | get,list |
| Cluster:default/sveltos-management-workload | hr | human-resource | * | * | * | * |
+---------------------------------------------+-------+-----------------+------------+-----------+----------------+----------+
If eng admin tries to deploy nginx in eng namespace, that will succeed.
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx
namespace: default
data:
nginx.yaml: |
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: eng
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
---
apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
name: deploy-nginx
labels:
projectsveltos.io/admin-name: eng
spec:
clusterSelector: env=development
syncMode: Continuous
policyRefs:
- name: nginx
namespace: default
kind: ConfigMap
nginx is deployed
./bin/sveltosctl show features
+-------------------------------------+-----------------+-----------+------------------+---------+-------------------------------+------------------+
| CLUSTER | RESOURCE TYPE | NAMESPACE | NAME | VERSION | TIME | CLUSTER PROFILES |
+-------------------------------------+-----------------+-----------+------------------+---------+-------------------------------+------------------+
| default/sveltos-management-workload | apps:Deployment | eng | nginx-deployment | N/A | 2023-02-03 14:40:19 -0800 PST | deploy-nginx |
+-------------------------------------+-----------------+-----------+------------------+---------+-------------------------------+------------------+
Contribute
Sveltos is an open source project. Contributions are more welcome 🤗
If you would like to know more about it or would like to see a new feature added to sveltos, please reach to us on slack. Any feedback is very much appreciated ❤️
Documentation can be found here
Latest comments (1)
Sveltos is an open source project.
Contributions are more welcome 🤗
If you like golang and Kubernetes and would like to help, please contact us. We would love to have you part of the sveltos project.
Chat with us on the Slack in the #projectsveltos channel Slack