Learn Kubernetes - Create Multiple Schedulers

Howdy!

Most big tech companies provide great documentation on their software. Especially, have you looked at Kubernetes documentation? They are quite awesome.

Particularly, I love the examples that are given pretty much for any topic. You can just blindly follow them to understand Kubernetes concepts.

Today, I chose to learn about how to configure multiple schedulers in a Kubernetes cluster, of course from their documentation. I also wanted to use my Minikube cluster. Learn how to install Minikube here.

Start Minikube cluster.

%minikube start --driver=hyperkit
😄  minikube v1.25.2 on Darwin 12.4
✨  Using the hyperkit driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating hyperkit VM (CPUs=2, Memory=8192MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Find the image used by kube-scheduler pod running in your Minikube cluster.

%kubectl get pod kube-scheduler-minikube -o yaml -n kube-system | grep image
    image: k8s.gcr.io/kube-scheduler:v1.23.3
    imagePullPolicy: IfNotPresent
    image: k8s.gcr.io/kube-scheduler:v1.23.3
    imageID: docker-pullable://k8s.gcr.io/kube-scheduler@sha256:32308abe86f7415611ca86ee79dd0a73e74ebecb2f9e3eb85fc3a8e62f03d0e7

Create the YAML file for all the required resources

Here's the YAML file provided in Kubernetes documentation that creates all the resources required to create another scheduler. Since the second scheduler is also going to use the same image as the above, let's make sure to use this. What that means is to replace the image mentioned in the documentation with the above image.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-scheduler
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-scheduler-as-kube-scheduler
subjects:
- kind: ServiceAccount
  name: my-scheduler
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:kube-scheduler
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-scheduler-as-volume-scheduler
subjects:
- kind: ServiceAccount
  name: my-scheduler
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:volume-scheduler
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-scheduler-config
  namespace: kube-system
data:
  my-scheduler-config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1beta2
    kind: KubeSchedulerConfiguration
    profiles:
      - schedulerName: my-scheduler
    leaderElection:
      leaderElect: false    
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    component: scheduler
    tier: control-plane
  name: my-scheduler
  namespace: kube-system
spec:
  selector:
    matchLabels:
      component: scheduler
      tier: control-plane
  replicas: 1
  template:
    metadata:
      labels:
        component: scheduler
        tier: control-plane
        version: second
    spec:
      serviceAccountName: my-scheduler
      containers:
      - command:
        - /usr/local/bin/kube-scheduler
        - --config=/etc/kubernetes/my-scheduler/my-scheduler-config.yaml
        image: k8s.gcr.io/kube-scheduler:v1.23.3
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10259
            scheme: HTTPS
          initialDelaySeconds: 15
        name: kube-second-scheduler
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10259
            scheme: HTTPS
        resources:
          requests:
            cpu: '0.1'
        securityContext:
          privileged: false
        volumeMounts:
          - name: config-volume
            mountPath: /etc/kubernetes/my-scheduler
      hostNetwork: false
      hostPID: false
      volumes:
        - name: config-volume
          configMap:
            name: my-scheduler-config

Create all the required objects

%kubectl apply -f my-scheduler.yaml
serviceaccount/my-scheduler created
clusterrolebinding.rbac.authorization.k8s.io/my-scheduler-as-kube-scheduler created
clusterrolebinding.rbac.authorization.k8s.io/my-scheduler-as-volume-scheduler created
configmap/my-scheduler-config created
deployment.apps/my-scheduler created

Verify whether the new scheduler is created

%kubectl get deployment -n kube-system
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
coredns        1/1     1            1           23h
my-scheduler   1/1     1            1           8s

%kubectl get pods -n kube-system
NAME                               READY   STATUS    RESTARTS   AGE
coredns-64897985d-4vstq            1/1     Running   0          23h
etcd-minikube                      1/1     Running   0          23h
kube-apiserver-minikube            1/1     Running   0          23h
kube-controller-manager-minikube   1/1     Running   0          23h
kube-proxy-w4srd                   1/1     Running   0          23h
kube-scheduler-minikube            1/1     Running   0          23h
my-scheduler-5546b885fb-mdgcn      1/1     Running   0          18s
storage-provisioner                1/1     Running   0          23h

Create a pod but this time make it use the new scheduler

%cat pod-using-second-scheduler.yaml
apiVersion: v1
kind: Pod
metadata:
  name: annotation-second-scheduler
  labels:
    name: multischeduler-example
spec:
  schedulerName: my-scheduler
  containers:
  - name: pod-with-second-annotation-container
    image: k8s.gcr.io/pause:2.0

%kubectl apply -f pod-using-second-scheduler.yaml
pod/annotation-second-scheduler created

%kubectl get event
LAST SEEN   TYPE      REASON      OBJECT                            MESSAGE
19s         Normal    Scheduled   pod/annotation-second-scheduler   Successfully assigned default/annotation-second-scheduler to minikube
18s         Normal    Pulling     pod/annotation-second-scheduler   Pulling image "k8s.gcr.io/pause:2.0"
17s         Normal    Pulled      pod/annotation-second-scheduler   Successfully pulled image "k8s.gcr.io/pause:2.0" in 1.25784015s
17s         Normal    Created     pod/annotation-second-scheduler   Created container pod-with-second-annotation-container
17s         Normal    Started     pod/annotation-second-scheduler   Started container pod-with-second-annotation-container

That's it. Ta-daa !