Flomesh Ingress Controller with Kubernetes Multi-tenancy

Background

It's very rare for organizations to have or provide a dedicated Kubernetes cluster to each tenant, as it's generally more cost-effective to share a cluster. Sharing clusters reduces expenses and streamlines management. Sharing clusters, however, also comes with difficulties like managing noisy neighbors and ensuring security.

Clusters can be shared in many ways. In some cases, different applications may run in the same cluster. In other cases, multiple instances of the same application may run in the same cluster, one for each end user. All these types of sharing are frequently described using the umbrella term multi-tenancy.

While Kubernetes does not have first-class concepts of end users or tenants, it provides several features to help manage different tenancy requirements. These are discussed below.

• A common form of multi-tenancy is to share a cluster between multiple teams within an organization, each of whom may operate one or more workloads. In this scenario, members of the teams often have direct access to Kubernetes resources via tools such as kubectl, or indirect access through GitOps controllers or other types of release automation tools. There is often some level of trust between members of different teams, but Kubernetes policies such as RBAC, quotas, and network policies are essential to safely and fairly share clusters.

• The other major form of multi-tenancy frequently involves a Software-as-a-Service (SaaS) vendor running multiple instances of a workload for customers. This business model is so strongly associated with this deployment style that many people call it "SaaS tenancy." In this scenario, the customers do not have access to the cluster; Kubernetes is invisible from their perspective and is only used by the vendor to manage the workloads. Cost optimization is frequently a critical concern, and Kubernetes policies are used to ensure that the workloads are strongly isolated from each other.

Tenant Isolation

There are several ways to design and build multi-tenant solutions with Kubernetes at its control plane, data plane, or at both levels. Each of these methods comes with its own set of tradeoffs that impact the isolation level, implementation effort, operational complexity, and cost of service.

Kubernetes Control plane isolation ensures that different tenants cannot access or affect each other's Kubernetes API resources, and Flomesh Service Mesh (FSM) Ingress controller provides isolated Ingress controllers for Kubernetes Namespaces.

In Kubernetes, a Namespace provides a mechanism for isolating groups of API resources within a single cluster. This isolation has two key dimensions:

• Object names within a namespace can overlap with names in other namespaces, similar to files in folders. This allows tenants to name their resources without having to consider what other tenants are doing.

• Many Kubernetes security policies are scoped to namespaces. For example, RBAC Roles and Network Policies are namespace-scoped resources. Using RBAC, Users and Service Accounts can be restricted to a namespace.

In a multi-tenant environment, a Namespace helps segment a tenant's workload into a logical and distinct management unit. A common practice is to isolate every workload in its namespace, even if multiple workloads are operated by the same tenant. This ensures that each workload has its own identity and can be configured with an appropriate security policy.

In this article, we will learn how to use the Flomesh Service Mesh Ingress controller to create physical isolation of Ingress controllers when hosting multiple tenants in your Kubernetes cluster.

Flomesh Service Mesh (FSM)

FSM is an open-source product from Flomesh for Kubernetes north-south traffic, gateway API controller, and multi-cluster management. FSM uses Pipy, a programmable proxy at its core, and provides an Ingress controller, Gateway API controller, load balancer, cross-cluster service registration discovery, and more.

FSM Ingress Controller supports a multi-tenancy model via its concept of NamespacedIngress CRD where it deploys a physically isolated Ingress Controller for requested Namespace.

For example, the YAML below defines an Ingress Controller that monitors on port 100 and also creates a LoadBalancer type service for it that listens on port 100.

apiVersion: flomesh.io/v1alpha1
kind: NamespacedIngress
metadata:
  name: namespaced-ingress-100
  namespace: test-100
spec:
  serviceType: LoadBalancer
  ports:
  - name: http
    port: 100
    protocol: TCP

Install FSM

FSM provides a standard Helm chart, which can be installed via the Helm CLI.

$ helm repo add fsm https://flomesh-io.github.io/fsm
$ helm repo update

$ helm install fsm fsm/fsm --namespace flomesh --create-namespace --set fsm.ingress.namespaced=true

Verify that all pods are up and running properly.

$ kubectl get po -n flomesh
NAME                                          READY   STATUS    RESTARTS   AGE
fsm-manager-6857f96858-sjksm                  1/1     Running   0          55s
fsm-repo-59bbbfdc5f-w7vg6                     1/1     Running   0          55s
fsm-bootstrap-8576c5ff4f-7qr7k                1/1     Running   0          55s
fsm-cluster-connector-local-8f8fb87f6-h7z9j   1/1     Running   0          32s

Create Sample Application

In this demo, we will be deploying httpbin service under a namespace httpbin.

# Create Namespace
kubectl create ns httpbin

# Deploy sample
kubectl apply -f https://raw.githubusercontent.com/flomesh-io/osm-edge-docs/main/manifests/samples/httpbin/httpbin.yaml -n httpbin

Creating a standalone Ingress Controller

The next step is to create a separate Ingress Controller for the namespace httpbin.

$ kubectl apply -f - <<EOF
apiVersion: flomesh.io/v1alpha1
kind: NamespacedIngress
metadata:
  name: namespaced-ingress-httpbin
  namespace: httpbin
spec:
  serviceType: LoadBalancer
  http:
    port:
      name: http
      port: 81
      nodePort: 30081
  resources:
    limits:
      cpu: 500m
      memory: 200Mi
    requests:
      cpu: 100m
      memory: 20Mi
EOF

After executing the above command, you will see an Ingress Controller running successfully under the namespace httpbin.

kubectl get po -n httpbin -l app=fsm-ingress-pipy
NAME                                        READY   STATUS    RESTARTS   AGE
fsm-ingress-pipy-httpbin-5594ffcfcc-zl5gl   1/1     Running   0          58s

At this point, there should be a corresponding Service under this namespace.

$ kubectl get svc -n httpbin -l app.kubernetes.io/component=controller
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)        AGE
fsm-ingress-pipy-httpbin   LoadBalancer   10.43.62.120   192.168.1.11   81:30081/TCP   2m49s

Once you have the Ingress Controller, it's time to create the Ingress resource.

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  namespace: httpbin
spec:
  ingressClassName: pipy
  rules:
  - host: httpbin.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 14001
EOF

Now we have created Ingress resource, let's do a quick curl to see if things are working as expected.

For my local setup demo, LoadBalancer IP is 192.168.1.11, your IP might be different. So ensure you are performing a curl against your setup ExternalIP.

curl -sI http://192.168.1.11:81/get -H "Host: httpbin.org"
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Mon, 03 Oct 2022 12:02:04 GMT
content-type: application/json
content-length: 239
access-control-allow-origin: *
access-control-allow-credentials: true
connection: keep-alive

Conclusion

In this blog post, you learned about Kubernetes multi-tenancy, features provided by Kubernetes, tenancy isolation levels, and how to use Flomesh Service Mesh (FSM) Ingress controller to set up isolated Ingress controller for namespaces.

Flomesh Service Mesh(FSM) from Flomesh is Kubernetes North-South traffic manager, provides Ingress controllers, Gateway API, Load Balancer, and cross-cluster service registration and service discovery. FSM uses Pipy - a programmable network proxy, as its data plane and is suitable for cloud, edge, and IoT.