Liquid error: internal
Having been away from working on K8s for a while, I decided to cut open K8s to see how it works to try falling in love with it again. This is going to be a series of blogs, with the first surgery being trying to figure out what’s stored inside the etcd cluster.
This is a story of finding the etcd pod inside the kubernetes cluster without diving deep into how etcd works.
Etcd is kubernetes vault. It stores the entire state of the cluster: its configuration, specifications, and the statuses of the running workloads. In this blog,
What is etcd?
Etcd is defined as a distributed, reliable key-value store for the most critical data of a distributed system. Etcd is written in Go.
How does etcd work inside kubernetes?
Our friends at Heptio have a great blog where they have shown the components involved during a simple Pod creation process. It’s a great illustration of the API Server and etcd interaction.
Installing etcd
Download Binary
curl -L [https://github.com/etcd-io/etcd/releases/download/v3.3.11/etcdv3.3.11-linux-amd64.tar.gz](https://github.com/etcd-io/etcd/releases/download/v3.3.11/etcdv3.3.11-linux-amd64.tar.gz) -o etcd-v3.3.11-linux-amd64.tar.gz
Extract
tar xzvf etcd-v3.3.11-linux-amd64.tar.gz
Operate ETCD
Run server
./etcd
Run client (on a different terminal)
// Set a key
./etcdctl set key1 value1
// Get a key
./etcdctl get key1
One can also use etcd labs to play around with etcd deployment. It is similar to docker playground.
The Kubernetes Test Cluster
I use minikube along with a virtualboxdriver for starting a docker container, so the next task would be to setup up a one node K8s cluster on my local machine to interact with the etcd store.
Start the cluster
minikube start — vm-driver=virtualbox
Finding the etcd pod
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-5644d7b6d9-2qg54 1/1 Running 0 2d
coredns-5644d7b6d9-6mbqk 1/1 Running 0 2d
etcd-minikube 1/1 Running 0 2d
kube-addon-manager-minikube 1/1 Running 0 2d
kube-apiserver-minikube 1/1 Running 0 2d
kube-controller-manager-minikube 1/1 Running 0 20s
kube-proxy-kgt6n 1/1 Running 0 2d
kube-scheduler-minikube 1/1 Running 0 2d
storage-provisioner 1/1 Running 0 2d
Connecting to the etcd pod
kubectl exec -it etcd-minikube -n kube-system sh
After connecting to the shell, I tried running the etcdctl command to find all the keys.
etcdctl get /
Turns out, it wasn’t as simple as this. Etcd uses an ADVERTISE_URL over which it exposes the process. I had to figure out all the flags which etcd uses at startup. I used the following command to get all the necessary flags.
$ echo “$(ps aux)”
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 2.9 2.3 10611260 46916 ? Ssl 10:24 7:04
etcd --advertise-client-urls=[https://192.168.99.100:2379](https://192.168.99.100:2379) --cert-file=/var/lib/minikube/certs/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/minikube/etcd --initial-advertise-peer-urls=[https://192.168.99.100:2380](https://192.168.99.100:2380) --initial-cluster=minikube=[https://192.168.99.100:2380](https://192.168.99.100:2380) --key-file=/var/lib/minikube/certs/etcd/server.key --listen-client-urls=[https://127.0.0.1:2379,https://192.168.99.100:2379](https://127.0.0.1:2379,https://192.168.99.100:2379) --listen-metrics-urls=[http://127.0.0.1:2381](http://127.0.0.1:2381) --listen-peer-urls=[https://192.168.99.100:2380](https://192.168.99.100:2380) --name=minikube --peer-cert-file=/var/lib/minikube/certs/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/var/lib/minikube/certs/etcd/peer.key --peer-trusted-ca-file=/var/lib/minikube/certs/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/var/lib/minikube/certs/etcd/ca.crt
To get etcdctl to talk to the etcd server, it needs the above flags since K8s sets up etcd along with authentication inside the pod. We first need to figure out the ADVERTISE_URL, which can be found using the --advertise-client-urls flag.
Getting all the keys
$ ADVERTISE_URL=”[https://192.168.99.100:2379](https://192.168.99.100:2379)"
$ ETCDCTL_API=3 etcdctl --endpoints $ADVERTISE_URL \
--cacert /var/lib/minikube/certs/etcd/ca.crt \
--cert /var/lib/minikube/certs/etcd/server.crt \
--key /var/lib/minikube/certs/etcd/server.key \
get / — prefix — keys-only
Funny output
/registry/apiregistration.k8s.io/apiservices/v1.
/registry/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.apps
/registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.autoscaling
/registry/apiregistration.k8s.io/apiservices/v1.batch
/registry/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.networking.k8s.io
/registry/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io
Let’s make sense of this by storing the output it in a json file.
$ ETCDCTL_API=3 etcdctl --endpoints $ADVERTISE_URL \
--cacert /var/lib/minikube/certs/etcd/ca.crt \
--cert /var/lib/minikube/certs/etcd/server.crt \
--key /var/lib/minikube/certs/etcd/server.key \
get / — prefix — keys-only -w json > out.json
It outputs all the KV pairs in a json format where they keys are encoded.
{
"header": {
"cluster_id": 12197035334886545600,
"member_id": 9217530203749069991,
"revision": 19300,
"raft_term": 2
},
"kvs": [{
"key": "L3JlZ2lzdHJ5L2FwaXJlZ2lzdHJhdGlvbi5rOHMuaW8vYXBpc2VydmljZXMvdjEu",
"create_revision": 12,
"mod_revision": 12,
"version": 1
}, {
"key": "L3JlZ2lzdHJ5L2FwaXJlZ2lzdHJhdGlvbi5rOHMuaW8vYXBpc2VydmljZXMvdjEuYWRtaXNzaW9ucmVnaXN0cmF0aW9uLms4cy5pbw==",
"create_revision": 9,
"mod_revision": 9,
"version": 1
}, {
"key": "L3JlZ2lzdHJ5L2FwaXJlZ2lzdHJhdGlvbi5rOHMuaW8vYXBpc2VydmljZXMvdjEuYXBpZXh0ZW5zaW9ucy5rOHMuaW8=",
"create_revision": 10,
"mod_revision": 10,
"version": 1
}]
}
Output for one key
$ ETCDCTL_API=3 etcdctl — endpoints $ADVERTISE_URL — cacert /var/lib/minikube/certs/etcd/ca.crt — cert /var/lib/minikube/certs/etcd/server.crt — key /var/lib/minikube/certs/etcd/server.key get /registry/apiregistration.k8s.io/apiservices/v1.
{
"kind": "APIService",
"apiVersion": "apiregistration.k8s.io/v1beta1",
"metadata": {
"name": "v1.",
"uid": "fd15144e-ef8c-4a02–87fc-0fc72c178118",
"creationTimestamp": "2019–11–22T15:19:43Z",
"labels": {
"kube-aggregator.kubernetes.io/automanaged": "onstart"
}
},
"spec": {
"service": null,
"version": "v1",
"groupPriorityMinimum": 18000,
"versionPriority": 1
},
"status": {
"conditions": [{
"type": "Available",
"status": "True",
"lastTransitionTime": "2019–11–22T15:19:43Z",
"reason": "Local",
"message": "Local APIServices are always available"
}]
}
}
The above keys are the configuration and status of all the resources within the cluster like:
Nodes
Namespaces
ClusterRoles
ClusterRoleBindings
ConfigMaps
Secrets
Workloads: Deployments, DaemonSets, Pods
Summary
This is my personal log on how to connect to the etcd pod inside a kubernetes cluster for debugging purpose in the future. It is interesting to see what information a K8s cluster stores in etcd when running. Helps in learning how to design complex systems in the future.
Please follow TechLog for more.
Top comments (0)