How to monitor Openshift using Datadog Operator

Co-authors: Luiz Bernardo Levenhagen and Leonardo Araujo

In this article, we will demonstrate how to integrate Openshift with Datadog using Datadog operator to collect metrics,logs, events and also applications' data.

In this article we use the following versions:

• Openshift v4.13.11
• Datadog Operator v1.3.0
• Datadog account (more information on how to request a trial at the bottom of the blog)

About

• This article is aimed at users who would like to integrate or monitor their Openshift Cluster using the Datadog monitoring solution.

• We will use the datadog operator to instantiate our agent and collect all metrics(cluster/containers), cluster and container/pod logs, network, cpu, memory consumption as well as applications' data.

• Red Hat does not support the DataDog operator or its configuration, for any questions related to the use of the platform or operator, contact DataDog.

Prerequisites

• User with the cluster-admin cluster role
• Openshift 4.10 or +
• Datadog account (more information on how to request a trial at the bottom of the blog)

Procedure

Datadog

Add API Keys

• To add a new datadog API Key, navigate to Organization Settings > API Keys
• If you have the permission to create API keys, click New Key in the top right corner.
• Define the desired name, something that can help you identify in the future.
• Once created, copy the Key so we can use it later.

Add Application keys

• To add a new datadog Application Key, navigate to Organization Settings > Application Keys
• If you have the permission to create Application Keys, click New Key in the top right corner.
• Define the desired name, something that can help you identify in the future.
• Once created, copy the Key so we can use it later.

Openshift

Datadog Operator Install

• In the Openshift console, in the left side menu, click Operator > OperatorHub > in the search field, type datadog

💡 Tip
Whenever available, use a certified option.

• As we can see, we are using version 1.3.0 of operator, click Install.

• On this screen, we will keep all the default options:
  • Update channel: stable
  • Installation mode: All namespaces the cluster(default)
  • Installed Namespace: openshift-operators
  • Update approval: Automatic
    • Obs.: If you prefer, you can use the Manual option.
  • Click Install.

• Wait until the installation is complete.

Create secret with Datadog keys (not mandatory, but good practice)

• In the terminal, access the openshift-operators namespace context

$ oc project openshift-operators

• Now let's create a secret to store in this API Key and Application Key, replace the values below with the keys we generated previously in the Datadog console.

$ oc create secret generic datadog-secret \
--from-literal api-key=`REPLACE_ME` \
--from-literal app-key=`REPLACE_ME`

• Let's now instatiate our datadog agent using the yaml below

$ cat <<EOF > datadog_agent.yaml
apiVersion: datadoghq.com/v2alpha1
kind: DatadogAgent
metadata:
  name: datadog
  namespace: openshift-operators
spec:
  features:
    apm:
      enabled: true
      unixDomainSocketConfig:
        enabled: true
    clusterChecks:
      enabled: true
      useClusterChecksRunners: true
    dogstatsd:
      originDetectionEnabled: true
      unixDomainSocketConfig:
        enabled: true
    eventCollection:
      collectKubernetesEvents: true
    liveContainerCollection:
      enabled: true
    liveProcessCollection:
      enabled: true
    logCollection:
      containerCollectAll: true
      enabled: true
    npm:
      collectDNSStats: true
      enableConntrack: true
      enabled: true
  global:
    clusterName: DemoLab
    credentials:
      apiSecret:
        keyName: api-key
        secretName: datadog-secret
      appSecret:
        keyName: app-key
        secretName: datadog-secret
    criSocketPath: /var/run/crio/crio.sock
    kubelet:
      tlsVerify: false
    site: datadoghq.eu
  override:
    clusterAgent:
      containers:
        cluster-agent:
          securityContext:
            readOnlyRootFilesystem: false
      replicas: 2
      serviceAccountName: datadog-agent-scc
    nodeAgent:
      hostNetwork: true
      securityContext:
        runAsUser: 0
        seLinuxOptions:
          level: s0
          role: system_r
          type: spc_t
          user: system_u
      serviceAccountName: datadog-agent-scc
      tolerations:
      - operator: Exists
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
EOF        

• Some explanations about what we are enabling in this agent

Enabling the APM (Application Performance Monitoring) feature

apm:
  enabled: true
  unixDomainSocketConfig:
    enabled: true

Cluster Check extends the autodiscover function to non-containerized resources and checks if there is some integration/technology to monitor.

clusterChecks:
  enabled: true
  useClusterChecksRunners: true

Dogstatsd is responsible for collecting custom metrics and events and sending them from time to time to a metrics aggregation service on the Datadog server.

dogstatsd:
  originDetectionEnabled: true
  unixDomainSocketConfig:
    enabled: true

Here we are enabling the collection of all logs (including container logs) and events generated in our cluster and sending them to Datadog.

eventCollection:
  collectKubernetesEvents: true
liveContainerCollection:
  enabled: true
liveProcessCollection:
  enabled: true
logCollection:
  containerCollectAll: true
  enabled: true

With NPM (Network Performance Monitoring), we can have visibility of all traffic in our cluster, nodes, containers, availability zones, etc.

npm:
  collectDNSStats: true
  enableConntrack: true
  enabled: true

In the credentials block in Global, we have the definition of the secret previously created with the API and app key.

credentials:
  apiSecret:
    keyName: api-key
    secretName: datadog-secret
  appSecret:
    keyName: app-key
    secretName: datadog-secret

In this block, we define the path to the cri-o service socket, we define the non-checking of tls for communication with the kubelet and in website, we define which datadog server will receive the data sent.

criSocketPath: /var/run/crio/crio.sock
kubelet:
  tlsVerify: false
site: datadoghq.eu

In the clusterAgent block in override, we add SecurityContext(scc) settings and which serviceaccount should be used in the datadog-cluster-agent pods.

clusterAgent:
  containers:
    cluster-agent:
      securityContext:
        readOnlyRootFilesystem: false
  replicas: 2
  serviceAccountName: datadog-agent-scc

❗Note
The datadog-agent-scc serviceaccount is created automatically by the operator and already has all the necessary permissions for the agent to run correctly.

In the nodeAgent block in override, we define settings for SecurityContext for the datadog-agent pods, we will use the same datadog-agent-scc serviceaccount and we also define the tolerations for the nodes that have taints created, in our case for the master nodes.

nodeAgent:
  hostNetwork: true
  securityContext:
    runAsUser: 0
    seLinuxOptions:
      level: s0
      role: system_r
      type: spc_t
      user: system_u
  serviceAccountName: datadog-agent-scc
  tolerations:
  - operator: Exists
  - effect: NoSchedule
    key: node-role.kubernetes.io/master

• After some explanations, let's deploy our datadog agent. Execute this command to create the object:

$ oc -n openshift-operators create -f datadog_agent.yaml

• Once created, we will validate that our agent was created correctly

$ oc -n openshift-operators get datadogagent
$ oc -n openshift-operators get pods

❗Note
Here we should have a datadog-agent running on each available openshift node.

❗Information

• datadog-agent-xxxxx pods, is responsible for collecting all metrics, events, traces and logs from each node in the cluster.
• datadog-cluster-agent-xxxxx pods, will act as a proxy between the API server and node-based agents, Cluster Agent helps to ease the server load.

• Now let's validate the logs of the datadog-agent-xxxxx pods, to identify if there is any communication error.

$ oc logs -f -l app.kubernetes.io/managed-by=datadog-operator --max-log-requests 10

Datadog platform/UI

• Now on the Datadog platform, in the left side menu, click on Infrastructure > and then onInfrastructure List`

❗Information
Server data, such as status, cpu information, memory and other details, may take a few minutes to be displayed.

• To view more details about a specific node, click on the node name and navigate through the available tabs. It’s just the simplest way to check your nodes/hosts.

• Under the Infrastructure menu, Datadog also gives you an exclusive Kubernetes menu where you have the full picture about your cluster. You can check the state of all of your Kubernetes resources, troubleshoot patterns, access out-of-the-box Dashboards and enable some recommended Alerts to monitor your environment

• You can also explore deeper the containers running in your Openshift environment, going to Infrastructure > Containers. Here you get chance to analyse things like logs from containers, traces, networking layer, processes running inside the container and so on...

• To view more details about network traffic, in the left side menu, go to Infrastructure > Network Map`

• To view the logs received from the cluster or from any application or technology running in your kubernetes environment, in the left side menu, go to Logs > Analytics, on this screen, we can view all the details, filter application logs and even view the processes.

• To view all collected metrics, in the left side menu, go to Metrics >Explorer`, here we can view all metrics, run and save queries or create dashboards based on queries.

• Datadog provides out-of-the-box Dashboards that can be used and customized. To use one available, in the left side menu, go to Dashboards > Dashboard List` > choose the dashboard and click on the name.

❗Note:
To customize a dashboard provided by Datadog, use the Clone feature to make the desired changes and save.

Conclusion

Using the Datadog Operator solution, we can have a complete monitoring solution for our Openshift cluster with main features such as APM, Network Analysis, Logs, Events and Metrics.

To request an Openshift trial and learn more about our solution, click here.

To request a Datadog trial and be able to replicate this knowledge, click here.

References

For more details and other configurations, start with the reference documents below.

• Red Hat Catalog - Datadog Operator
• Datadog Documentation - Openshift
• Datadog Documentation - Advanced setup for Datadog Operator
• Custom Datadog SCC for all features