Using logrotate to manage Apama Streaming Analytics log files

Introduction

Apama applications will frequently produce large log files, both from the platform itself and from applications. Apama provides a built-in log-rotation mechanism to cover most use cases. For details of the built-in mechanism see “Rotating correlator log files” in the Apama documentation.

For some use cases, however, the Apama mechanism is not sufficient, and an external tool is needed. The most common such tool is the Linux logrotate tool. This is a very powerful and flexible tool which allows you to configure a large range of rotation policies and Apama is fully compatible with it.

Pre-requisite

To use logrotate you will need an Apama project which is configured to generate a pidfile and one or more logfiles. By default an Apama project will generate both of these in the logs directory of a deployed project.

This is normally most applicable to standalone Apama running on-premise, since many cloud deployments will be configured to log to stdout and use the cloud log management tools. However, we will use a Docker deployment to demonstrate the capability here to allow people to most easily reproduce the setup and test it out.

Steps to follow

First, you will need to create a logrotate configuration file for Apama. This is where you define all of the policy on how often log rotate will rotate the log files, how many it will keep and so on. Log rotate can do time- or size-based rotation. We have decided here to rotate the log every day, keep the last 14 days log files and compress any logs older than the most recent two. Create a file called “apama.logrotate” in your project with the following contents:

/apama_work/deployed/logs/*.log {
    rotate 14
    daily
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
    kill -HUP `cat /apama_work/deployed/logs/defaultCorrelator.pidfile` || true
    endscript
}

The file also contains a script to run after logrotate moves the old log files to cause Apama to create the new set of log files.

For more details on what options are available to logrotate, see the logrotate manual page with man logrotate.

Next, when creating your Docker image containing your project, the resulting image will need to install logrotate. You can do this by adding the following lines to the Dockerfile created for your project. In a multi-stage build these will need to be added to the final stage:

USER root
RUN microdnf install -y logrotate && microdnf clean all
USER sagadmin

For an on-premise installation logrotate will already be installed, and so you can skip this step.

A full example of that Docker file would look like:

ARG APAMA_VERSION=10.15                                                                                                                   
ARG APAMA_BUILDER=softwareag/apama-builder:${APAMA_VERSION}                                                                                                 
ARG APAMA_IMAGE=softwareag/apama-correlator:${APAMA_VERSION}     
FROM ${APAMA_BUILDER} as builder                                                                                                                            
COPY --chown=sagadmin:sagadmin . ${APAMA_WORK}/Project
RUN engine_deploy --outputDeployDir ${APAMA_WORK}/Project_deployed ${APAMA_WORK}/Project
FROM ${APAMA_IMAGE}                                                                                                                             
USER root
RUN microdnf install -y logrotate && microdnf clean all
USER sagadmin
COPY --chown=sagadmin:sagadmin --from=builder ${APAMA_WORK}/Project_deployed ${APAMA_WORK}/Project_deployed
WORKDIR ${APAMA_WORK}          
CMD ["correlator", "--config", "Project_deployed"]

Build and deploy your Docker container as normal

Rotating the log files

To trigger the rotation, you need to run logrotate on your configuration file. For this Docker-based deployment we can do this by running logrotate manually with a Docker exec:

docker exec apamacontainer logrotate /apama_work/Project_deployed/apama.logrotate -s /apama_work/logrotate.state

Replace apamacontainer with the name of your running container.

This command can be run as often as you like and logrotate will determine when to rotate from the configuration file. If you want to force a rotation, then also provide the --force flag.

For an on-premise installation then logrotate will run automatically via the cron tool. To rotate the Apama logs just copy the logrotate configuration into the /etc/logrotate.d/ directory.

Results

The result of several rotations with the above configuration will result in /apama_work/Project_deployed/logs/ looking like:

correlator_defaultCorrelator.log
correlator_defaultCorrelator.log.1
correlator_defaultCorrelator.log.2.gz

The current log file is the .log, and older ones have sequential numbers starting at 1. All the logs after the current and the first rotation are also compressed for space. If your application generates multiple log files (for example, for different application packages), each one will be named in this fashion.

NOTE: If you are using logrotate it’s important not to use any of the Apama rotation identifiers or timestamps in the log file names, otherwise logrotate will not be able to identify the rotated file as being the same log as the previous one.

Useful links | Relevant resources

The Apama documentation can be found at Software AG Product Documentation

The logrotate documentation can be found at logrotate(8) - Linux manual page (man7.org)

Read full topic