Custom App Logging with Azure Monitor for Containers

Table of Contents

• Introduction
  • Our Scenario
  • Objective
  • Solution
• Implementation
  • Prerequisites
  • Step 1 - Invoking syslog in a PHP application
  • Step 2 - Dockerfile configurations
  • Step 3 - Docker CLI commands
  • Step 4 - Deploy to AKS
• In Closing
• Resources
• Acknowledgements

Introduction

If you use Azure, you probably spend time in Azure Monitor - Azure's comprehensive solution for collecting, analyzing, and acting on telemetry from your cloud and on-premises environments.

And, if you happen to run containerized workloads in Azure - say with Azure Kubernetes Service (AKS) or Azure Container instances (ACI) - you may already be tapping into Azure Monitor for containers. If you are new to this feature, in short...

Azure Monitor for containers gives you performance visibility by collecting memory and processor metrics from controllers, nodes, and containers that are available in Kubernetes through the Metrics API. Container logs are also collected.

For emphasis, I'll now repeat that last sentence...

Container logs are also collected.

Azure Monitor for containers collects stdout and stderr from container workloads deployed to AKS (or ACI).

Knowing this, all one needs to do is route custom application logs to stderr (or stdout) to take advantage of Azure Monitor for containers.

In this post, I will walk you through an example implementation using a simple php application and Rsyslog, the rocket-fast system for log processing.

Our Scenario

We have a containerized PHP application running in Azure Kubernetes Service (AKS). In our application, we leverage the syslog php function to generate custom log messages, which in turn are consumed by a user defined log handler.

Objective

Collect the log messages in Azure Monitor for containers for real-time observation and analysis in a Log Analytics workspace.

Below are screen captures from Azure Portal that show the expected custom log output in:

• The Live Data (preview) feature (Figure 1)
• A Log Analytics workspace (Figure 2)
• The Logs displayed for our container running in Azure Container Instances (Figure 3)

Figure 1

Figure 2

Figure 3

Solution

In our container image we install the rsyslog library; configure a user defined syslog log handler; and route the messages to stderr.

Syslog recognizes the following severity levels for log messages:

In our example implementation, we will evaluate for two severity levels: error (3) and info (6).

The implementation steps below are taken from this GitHub repo to which I contributed

Implementation

Prerequisites

This article assumes experience with Docker containers, Kubernetes and Azure. This article does not describe the steps to install an AKS cluster. Links to a Quickstart tutorial is provided in Resources.

This article uses a simple PHP application for the implementation. If you're not a PHP programmer, don't despair - the principals covered here are generally applicable to other languages.

Please note that our image operating system is from the image debian:stretch-slim.

Step 1 - Invoking syslog in a PHP application

The following code example is a simple, single page application I spun up for a "custom application logging with syslog demo". For the demo,
query strings are used to trigger the log handler, such as:

?value=0 // Triggers an Error log as a result of division by zero.
?value=2 // Triggers an Information log.

The application code:

<?php
echo '<H1>Syslog Demo</H1>';

// Fetch the URL query string and assign to a variable. 
$qstring = $_SERVER['QUERY_STRING'];

parse_str($qstring, $output);

$value = $output['value'];

// For an error evaluation, we divide 1 by $value.
// A value of 0 will throw a division by zero error.
function inverse($x) {
    if (!$x) {
        throw new Exception('Division by zero.');
    }
    return 1/$x;
}

// Get formatted datetime to include in messages.
$access = date("Y/m/d H:i:s");

try {
    echo "<p>Inverse of value " , $value , " is " , inverse($value) , "</p>";
    $message = "Inverse of value succeeded";
    $severity = LOG_INFO;
    $logtext = "INFORMATION: at " . $access . "\n" . $message . "\n" . $_SERVER['REMOTE_ADDR'];
    _log($severity, $logtext);

} catch (Exception $e) {
    $message = $e->getMessage();
    echo '<p>Caught exception: ',  $message, "</p>";
    $severity = LOG_ERR;
    $logtext = "ERROR: at " . $access . "\n" . $message . "\n" . $_SERVER['REMOTE_ADDR'];
    _log($severity, $logtext);
}

// Open and close connection of system logger.
function _log($priority, $text) {
  openlog("myApp", LOG_PID | LOG_PERROR, LOG_LOCAL0);  
  syslog($priority, $text);
  closelog();
}

?>

Step 2 - Dockerfile configurations

To enable syslog support in our container image, we install the rsyslog library:

RUN apt-get update; \
    apt-get install -y --no-install-recommends \
    rsyslog \
    ;

Later in our Dockerfile, we configure a custom log handler for our application logs:

RUN echo "local0.* /var/log/apache2/myapp.log" >> /etc/rsyslog.conf

We then configure the log to go to stderr using a symlink, and we re-set ownership of the /var/log/apache2/ just as first set in our base image php:7.3-apache-stretch:

RUN \ 
  ln -sfT /dev/stderr "/var/log/apache2/myapp.log"; \
  chown -R --no-dereference "www-data:www-data" "/var/log/apache2/"

Step 3 - Docker CLI commands

Build a container image:

docker build -t applogdemo .

Test it locally:

docker run --name applogdemo --rm -i -t applogdemo

The log output local test results should look similar to:

[ ok ] Starting enhanced syslogd: rsyslogd.
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
[Sat Jun 20 20:34:48.662731 2020] [mpm_prefork:notice] [pid 40] AH00163: Apache/2.4.25 (Debian) PHP/7.3.13 configured -- resuming normal operations
[Sat Jun 20 20:34:48.662809 2020] [core:notice] [pid 40] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND'

While the container is running, in your browser enter some URL's to invoke the log handler, such as:
http://localhost/?value=0, http://localhost/?value=2, etc.

You should see additional log output similar to:

myApp[41]: ERROR: at 2020/06/29 14:18:15
myApp[41]: Division by zero.
myApp[41]: 172.17.0.1
172.17.0.2:80 172.17.0.1 - - [29/Jun/2020:14:18:15 +0000] "GET /?value=0 HTTP/1.1" 200 366 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:7
7.0) Gecko/20100101 Firefox/77.0"
"-" - - [29/Jun/2020:14:18:15 +0000] "GET /?value=0 HTTP/1.1" 200 366 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 F
irefox/77.0"
myApp[42]: INFORMATION: at 2020/06/29 14:18:30
myApp[42]: Inverse of value succeeded
myApp[42]: 172.17.0.1
172.17.0.2:80 172.17.0.1 - - [29/Jun/2020:14:18:30 +0000] "GET /?value=4 HTTP/1.1" 200 268 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:7
7.0) Gecko/20100101 Firefox/77.0"
"-" - - [29/Jun/2020:14:18:30 +0000] "GET /?value=4 HTTP/1.1" 200 268 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0"

Now that we have validated the application locally, let's prep the image for publishing to our preferred container registry, Azure Container Registry (Docker Hub or other registry is fine). Let's tag the image:

docker tag applogdemo myacr.azurecr.io/applogdemo:v1

Let's push to our preferred container registry.

docker push myacr.azurecr.io/applogdemo:v1

Step 4 - Deploy to AKS

To deploy the container, we use the following Kubernetes YAML manifest.

Note the inclusion of a Service resource.

If you are interested in deploying to Azure Container Instances, please refer to the Quickstart tutorial link provided in Resources.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: applogdemo
  labels:
    app: applogdemo
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: applogdemo
  template:
    metadata:
      labels:
        app: applogdemo
    spec:
      containers:
        - image: myacr.azurecr.io/applogdemo:v1 // Placeholder
          name: applogdemo
          ports:
          - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: applogdemo-service
  namespace: default
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
  selector:
    app: applogdemo
  type: LoadBalancer

Customize the image: value in the containers: spec if the deployment.yml manifest, e.g.

containers:
    -
        image: myacr.azurecr.io/applogdemo:v1

Finally, deploy the kubernetes manifests using the commands:

   kubectl apply -f manifests/deployment.yml

Step 5 - Validate the deployment

Validate the deployment by accessing the website via the IP Address exposed by the Kubernetes LoadBalancer service. To identify the IP Address, use the command:

$ kubectl get svc drupal-service
NAME             TYPE           CLUSTER-IP   EXTERNAL-IP      PORT(S)        AGE
drupal-service   LoadBalancer   10.2.0.50    52.151.xxx.xxx   80:32758/TCP   10d

Test the application just as done for local testing.
Once you have called a sequence of URL's with the ?value=<some integer> query string go to the Azure Monitor for containers blade in Azure Portal to view your custom application logs.

In Closing

In this article we have covered the basic configurations necessary to enable custom application logging for Azure Monitor for containers.

From a log analysis standpoint, we've only scratched the surface in this post. Explore the Kusto query language to craft a wide variety of reports. Create alerts based on log metrics. Pin and share custom log dashboards.

Please feel free to share your questions and comments in the discussion below. Thanks - Mike 😃

📚 Resources

• Azure Monitor for containers overview
• Getting started with Kusto
• Quickstart: Deploy an Azure Kubernetes Service cluster using the Azure CLI
• Quickstart: Deploy a container instance in Azure using the Azure CLI
• Rsyslog
• PHP syslog function
• Log rotation with rsyslog
• Docker image code for php:7.3-apache-stretch

👏 Acknowledgements

I would like to thank Firoz Shaik for his review of this post and insightful recommendations.