There are various ways to capture logs in kubernetes. One of the simplest mechanism is to have pod level script that can upload logs to destination system. This approach can work when you are trying out kubernetes but as soon as you decide to use kubernetes in production and have multiple applications to deploy, you see a need of having a centralized logging mechanism which is independent of pod lifecycle.
We had the requirement to set up logging for our kubernetes cluster. Cluster is built on Oracle Kubernetes Engine (OKE) and we wanted to persist logs in OCI Object Storage. First step was to find out an efficient way of capturing logs. Looking into multiple options, we found Daemonset to be a great option due to following reasons:
- Kubernetes ensures that every node runs a copy of Pod. Any new node gets added in the cluster, kubernetes automatically ensures to bring up a pod on the node. This can be further customized to only choose nodes based on selection criteria.
- It avoids changing individual application deployment. If later you want to change your logging mechanism, you only need to change your Daemonset
- There is no impact on performance of application due to log capturing as it’s running outside pod.
Once finalized, next step was to configure Daemonset to capture logs from OKE and publish them to OCI Object Storage. We have used fluentd before and we decided to go ahead with it. Fluentd already have image for configuring daemonset and upload to s3. Since object storage is compatible with S3 API, we were able to use it with some customizations of fluent.conf.
Setting up cluster role
Fluentd daemonset requires to run in kube-system. First step is to create a new account and providing it required privileges.
Create Service Account:
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluentd
namespace: kube-system
kubectl create -f serviceaccount.yaml
Create Cluster Role:
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: fluentd
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- pods
- namespaces
verbs:
- get
- list
- watch
kubectl create -f clusterrole.yaml
Create binding for cluster role with account:
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: fluentd
roleRef:
kind: ClusterRole
name: fluentd
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: fluentd
namespace: kube-system
kubectl create -f clusterrolebinding.yaml
Create Daemonset
Next, we create config map to provide custom fluent configuration:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-config
namespace: kube-system
data:
fluent.conf: |
@include "#{ENV['FLUENTD_SYSTEMD_CONF'] || '../systemd'}.conf"
@include "#{ENV['FLUENTD_PROMETHEUS_CONF'] || '../prometheus'}.conf"
@include ../kubernetes.conf
@include conf.d/*.conf
<match **>
@type s3
@id out_s3
@log_level info
s3_bucket "#{ENV['S3_BUCKET_NAME']}"
s3_endpoint "#{ENV['S3_ENDPOINT']}"
s3_region "#{ENV['S3_BUCKET_REGION']}"
s3_object_key_format %{path}%Y/%m/%d/cluster-log-%{index}.%{file_extension}
<inject>
time_key time
tag_key tag
localtime false
</inject>
<buffer>
@type file
path /var/log/fluentd-buffers/s3.buffer
timekey 3600
timekey_use_utc true
chunk_limit_size 256m
</buffer>
</match>
kubectl create -f configmap.yaml
Now, we can go ahead and create daemonset:
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: fluentd
namespace: kube-system
labels:
k8s-app: fluentd-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
template:
metadata:
labels:
k8s-app: fluentd-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
serviceAccount: fluentd
serviceAccountName: fluentd
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:v1.3.3-debian-s3-1.3
env:
- name: AWS_ACCESS_KEY_ID
value: "#{OCI_ACCESS_KEY}"
- name: AWS_SECRET_ACCESS_KEY
value: "#{OCI_ACCESS_SECRET}"
- name: S3_BUCKET_NAME
value: "#{BUCKET_NAME}"
- name: S3_BUCKET_REGION
value: "#{OCI_REGION}"
- name: S3_ENDPOINT
value: "#{OBJECT_STORAGE_END_POINT}"
- name: FLUENT_UID
value: "0"
- name: FLUENTD_CONF
value: "override/fluent.conf"
- name: FLUENTD_SYSTEMD_CONF
value: "disable"
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log/
- name: u01data
mountPath: /u01/data/docker/containers/
readOnly: true
- name: fluentconfig
mountPath: /fluentd/etc/override/
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log/
- name: u01data
hostPath:
path: /u01/data/docker/containers/
- name: fluentconfig
configMap:
name: fluent-config
Couple of things to note:
- We provide a custom fluent.conf via config map and mount it in daemonset. This is required to provide explicit s3_endpoint since image by default doesn’t have a way to provide custom s3_endpoint
- Following are the env variables we need to configure. S3_BUCKET_REGION is the oci region e.g. us-ashburn-1. S3_ENDPOINT is the object storage endpoint e.g. https://#{tenantname}.compat.objectstorage.us-ashburn-1.oraclecloud.com. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are the customer secret keys for your user. If not already present, refer to doc to generate. S3_BUCKET_NAME is the object storage bucket to store logs.
Let’s go ahead and create daemonset:
kubectl create -f daemonset.yaml
Once configured, you should be able to see logs in object storage. If bucket doesn’t exist, it will create it.
Top comments (0)