Kubernetes event is report of an event happening within the cluster. It tracks any changes made to the cluster which is a good piece of information for debugging.
However, the retention time is limited.
Fluentbit
Fluent Bit is a fast and lightweight Telemetry agent. We will be using Kubernetes Events input plugin to process the Kubernetes event as log.
Monitor Kubernetes Event
First we will install Fluentbit using Helm. We will assume you already have a Kubernetes cluster handy.
We will deploy Fluentbit with a minimal configuration as shown below. Save this as values.yaml.
rbac:
eventsAccess: true
config:
inputs: |
[INPUT]
Name kubernetes_events
Tag k8s_events.*
kube_url https://kubernetes.default.svc
outputs: |
[OUTPUT]
Name stdout
Match k8s_events.*
We will need rbac.eventsAccess: true to give Fluentbit access to Kubernetes Event resource. In config.inputs, we define the input of Fluentbit which will be Kubernetes event in this case. And in config.outputs, we define where the output of the incoming Kubernetes event goes to.
$ helm repo add fluent https://fluent.github.io/helm-charts
$ helm repo update
$ helm upgrade --install fluent-bit fluent/fluent-bit -f values.yaml
Let's make sure that our Fluentbit is running.
kubectl get po
NAME READY STATUS RESTARTS AGE
fluent-bit-4dm2l 1/1 Running 0 27h
We got the fluentbit running so let's create a sample pod to create some Kubernetes events.
$ kubectl run nginx --image=nginx -- sleep 1000
And if we take a look at the fluent bit log, we can see the events related to nginx pod we just created.
$ kubectl logs ds/fluent-bit
[1] k8s_events.*: [[1694078558.000000000, {}], {"metadata"=>{"name"=>"nginx.17829379194ff602", "namespace"=>"default", "uid"=>"08314b94-2b02-4485-8a86-65cb07f1b3e1", "resourceVersion"=>"164604", "creationTimestamp"=>"2023-09-07T09:22:38Z", "managedFields"=>[{"manager"=>"kubelet", "operation"=>"Update", "apiVersion"=>"v1", "time"=>"2023-09-07T09:22:38Z", "fieldsType"=>"FieldsV1", "fieldsV1"=>{"f:count"=>{}, "f:firstTimestamp"=>{}, "f:involvedObject"=>{}, "f:lastTimestamp"=>{}, "f:message"=>{}, "f:reason"=>{}, "f:source"=>{"f:component"=>{}, "f:host"=>{}}, "f:type"=>{}}}]}, "involvedObject"=>{"kind"=>"Pod", "namespace"=>"default", "name"=>"nginx", "uid"=>"2efbf9c5-4155-48c1-add6-b8389c202e91", "apiVersion"=>"v1", "resourceVersion"=>"164599", "fieldPath"=>"spec.containers{nginx}"}, "reason"=>"Pulling", "message"=>"Pulling image "nginx"", "source"=>{"component"=>"kubelet", "host"=>"kind-control-plane"}, "firstTimestamp"=>"2023-09-07T09:22:38Z", "lastTimestamp"=>"2023-09-07T09:22:38Z", "count"=>1, "type"=>"Normal", "eventTime"=>nil, "reportingComponent"=>"", "reportingInstance"=>""}]
[0] k8s_events.*: [[1694078563.000000000, {}], {"metadata"=>{"name"=>"nginx.1782937a6e3d22e3", "namespace"=>"default", "uid"=>"2fe002bc-c1c5-47b5-b057-a0764602893a", "resourceVersion"=>"164611", "creationTimestamp"=>"2023-09-07T09:22:43Z", "managedFields"=>[{"manager"=>"kubelet", "operation"=>"Update", "apiVersion"=>"v1", "time"=>"2023-09-07T09:22:43Z", "fieldsType"=>"FieldsV1", "fieldsV1"=>{"f:count"=>{}, "f:firstTimestamp"=>{}, "f:involvedObject"=>{}, "f:lastTimestamp"=>{}, "f:message"=>{}, "f:reason"=>{}, "f:source"=>{"f:component"=>{}, "f:host"=>{}}, "f:type"=>{}}}]}, "involvedObject"=>{"kind"=>"Pod", "namespace"=>"default", "name"=>"nginx", "uid"=>"2efbf9c5-4155-48c1-add6-b8389c202e91", "apiVersion"=>"v1", "resourceVersion"=>"164599", "fieldPath"=>"spec.containers{nginx}"}, "reason"=>"Pulled", "message"=>"Successfully pulled image "nginx" in 5.71974592s (5.71977817s including waiting)", "source"=>{"component"=>"kubelet", "host"=>"kind-control-plane"}, "firstTimestamp"=>"2023-09-07T09:22:43Z", "lastTimestamp"=>"2023-09-07T09:22:43Z", "count"=>1, "type"=>"Normal", "eventTime"=>nil, "reportingComponent"=>"", "reportingInstance"=>""}]
[1] k8s_events.*: [[1694078563.000000000, {}], {"metadata"=>{"name"=>"nginx.1782937a6ea78c6c", "namespace"=>"default", "uid"=>"0058a999-0966-4b66-84c8-3361622fc7a8", "resourceVersion"=>"164612", "creationTimestamp"=>"2023-09-07T09:22:43Z", "managedFields"=>[{"manager"=>"kubelet", "operation"=>"Update", "apiVersion"=>"v1", "time"=>"2023-09-07T09:22:43Z", "fieldsType"=>"FieldsV1", "fieldsV1"=>{"f:count"=>{}, "f:firstTimestamp"=>{}, "f:involvedObject"=>{}, "f:lastTimestamp"=>{}, "f:message"=>{}, "f:reason"=>{}, "f:source"=>{"f:component"=>{}, "f:host"=>{}}, "f:type"=>{}}}]}, "involvedObject"=>{"kind"=>"Pod", "namespace"=>"default", "name"=>"nginx", "uid"=>"2efbf9c5-4155-48c1-add6-b8389c202e91", "apiVersion"=>"v1", "resourceVersion"=>"164599", "fieldPath"=>"spec.containers{nginx}"}, "reason"=>"Created", "message"=>"Created container nginx", "source"=>{"component"=>"kubelet", "host"=>"kind-control-plane"}, "firstTimestamp"=>"2023-09-07T09:22:43Z", "lastTimestamp"=>"2023-09-07T09:22:43Z", "count"=>1, "type"=>"Normal", "eventTime"=>nil, "reportingComponent"=>"", "reportingInstance"=>""}]
[2] k8s_events.*: [[1694078563.000000000, {}], {"metadata"=>{"name"=>"nginx.1782937a735969d5", "namespace"=>"default", "uid"=>"8830ae74-2f2c-4d5f-92a8-6f48aff7de59", "resourceVersion"=>"164615", "creationTimestamp"=>"2023-09-07T09:22:43Z", "managedFields"=>[{"manager"=>"kubelet", "operation"=>"Update", "apiVersion"=>"v1", "time"=>"2023-09-07T09:22:43Z", "fieldsType"=>"FieldsV1", "fieldsV1"=>{"f:count"=>{}, "f:firstTimestamp"=>{}, "f:involvedObject"=>{}, "f:lastTimestamp"=>{}, "f:message"=>{}, "f:reason"=>{}, "f:source"=>{"f:component"=>{}, "f:host"=>{}}, "f:type"=>{}}}]}, "involvedObject"=>{"kind"=>"Pod", "namespace"=>"default", "name"=>"nginx", "uid"=>"2efbf9c5-4155-48c1-add6-b8389c202e91", "apiVersion"=>"v1", "resourceVersion"=>"164599", "fieldPath"=>"spec.containers{nginx}"}, "reason"=>"Started", "message"=>"Started container nginx", "source"=>{"component"=>"kubelet", "host"=>"kind-control-plane"}, "firstTimestamp"=>"2023-09-07T09:22:43Z", "lastTimestamp"=>"2023-09-07T09:22:43Z", "count"=>1, "type"=>"Normal", "eventTime"=>nil, "reportingComponent"=>"", "reportingInstance"=>""}]
If we take a look at the Kubernetes event with kubectl, we can see the same event which we saw in the Fluentbit log for long term storage.
$ kubectl get events --field-selector involvedObject.name=nginx [18:31:21]
LAST SEEN TYPE REASON OBJECT MESSAGE
8m45s Normal Scheduled pod/nginx Successfully assigned default/nginx to kind-control-plane
8m44s Normal Pulling pod/nginx Pulling image "nginx"
8m39s Normal Pulled pod/nginx Successfully pulled image "nginx" in 5.71974592s (5.71977817s including waiting)
8m39s Normal Created pod/nginx Created container nginx
8m39s Normal Started pod/nginx Started container nginx
We tested out the Kubernetes event input plugin with a simple configuration. From here we can customize the fluentbit configuration to send the logs to third party storages outside of Kubernetes cluster .
Top comments (0)