Shipping logs from your Amazon EKS (Elastic Kubernetes Service) cluster to Amazon CloudWatch is a crucial part of maintaining visibility and observability in your Kubernetes environment. Fluent Bit, a lightweight and efficient log processor, is commonly used to collect, process, and forward logs to various destinations, including CloudWatch. This article will guide you through setting up Fluent Bit on your EKS cluster to send logs to CloudWatc
Step 1: Attach IAM policy to the EKS worker node role:
For FluenBit pods to ship logs from EKS nodes to CloudWatch, the nodes should have necessary permissions to perform CloudWatch operations.
Attach the “CloudWatchAgentServerPolicy” aws managed policy to the EKS worker node role
Step 2: Create amazon-cloudwatch namespace where fluent bit will be located.
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml
Step 3: Create ConfigMap for FluentBit using the below command:
ClusterName=cluster-name
RegionName=cluster-region
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
kubectl create configmap fluent-bit-cluster-info \
--from-literal=cluster.name=${ClusterName} \
--from-literal=http.server=${FluentBitHttpServer} \
--from-literal=http.port=${FluentBitHttpPort} \
--from-literal=read.head=${FluentBitReadFromHead} \
--from-literal=read.tail=${FluentBitReadFromTail} \
--from-literal=logs.region=${RegionName} -n amazon-cloudwatch
Step 4: Deploy the FluentBit as a daemonset:
In case the worker nodes are of linux based instances, use the below command to deploy the FluentBit daemonset:
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml
If the worker nodes are of windows based instances:
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit-windows.yaml
That’s it, now verify the FluentBit pods status:
kubectl get pods -n amazon-cloudwatch
Thank you for taking the time to read this article! Keep up the great work, and happy deploying! 🚀 😊
Top comments (0)