Dynatrace log alert: create an alert from logs and cut the noise

A Dynatrace log alert turns a log line into a Problem without anyone watching a console. The classic trap is too much or too little. An alert on every « error » line wakes the on-call engineer ten times for one cause. An alert that is too cautious misses the error that matters.

This article shows how to create the alert, then how to keep it quiet until something needs action. The main part applies to Dynatrace SaaS (OpenPipeline and Anomaly Detection). A separate section covers Dynatrace Managed, where the chain uses log metrics and metric events. The two platforms do not share the same screens, so I keep them apart.

In brief

• According to the Dynatrace SaaS documentation, there are three ways to build a Dynatrace log alert: Davis events extracted from logs, metrics extracted from logs with a custom alert, or a DQL query in a custom alert.
• The extracted event reacts fastest and needs no threshold. The extracted metric takes more setup but allows thresholds and analyzers.
• Noise is reduced in three places: the filter (matcher), the sliding window, and the choice not to alert on missing data for a sparse stream.

Choosing the right route for a Dynatrace log alert (SaaS)

The Alerting on logs page compares the three approaches. Here is the gist, in my own words.

The documentation states that the DQL query is not recommended as the main strategy: it runs every minute and can raise licence consumption. Keep it as a last resort.

For an SRE team, the rule is simple. A single serious event (« the export queue is full ») deserves the extracted event. A drift (« the error rate is climbing ») deserves a metric and a threshold. That choice sets the alert volume, and so the on-call toil.

Create the alert from an event extracted from logs

Prerequisites

Logs reach Dynatrace and you may configure OpenPipeline. The documentation cites the default « Data Processing and Storage » policy as an example. For ingestion itself, see the article on the Log ingestion API.

Step 1: isolate the logs with DQL

In Logs, write the query that finds the target line. The documentation example searches for an exact phrase.

fetch logs
| filter matchesPhrase(content, "Dropping data because sending_queue is full")
| sort timestamp desc

Open a record and read the dt.openpipeline.pipelines field: it tells you which pipeline processes that log.

Step 2: add the Davis event processor

1. Go to Settings > Process and contextualize > OpenPipeline > Logs > Pipelines, open the pipeline you need (or create one) and choose Edit.
2. In the Davis stage, add a Davis event processor.
3. In Matching condition, paste the filter extracted from your query.
4. In Event template, fill in the event properties.

matchesPhrase(content, "Dropping data because sending_queue is full")

The documentation template sets event.type to CUSTOM_ALERT, an event.name, an event.description and dt.source_entity. That last field, taken from the log, ties the alert to an entity and helps Davis root cause analysis.

Step 3: route the logs if the pipeline is new

If you edited an existing pipeline, nothing more is needed. Otherwise, create a route under OpenPipeline > Logs > Dynamic routing, with the same condition as the matcher. The documentation gives two rules. Routes are evaluated top to bottom, and the first match wins. Routing runs before processing, so a field added during processing, such as dt.entity.service, cannot be used in a route condition.

To go further with pipelines, read the Logs and OpenPipeline module.

Cutting the noise on the event side

One event per log line is a recipe for a flood. Three levers exist.

• A narrow matcher. Prefer a precise phrase to loglevel == "ERROR" alone. The finer the filter, the fewer events, and the documentation says cost follows the number of events generated.
• The timeout. According to the documentation, the dt.davis.event_timeout field sets how long the Problem stays open without a new event. The default is 15 minutes. While events keep arriving within that delay, the Problem stays the same; with no new event, it closes on its own.
• The right use. Keep this mechanism for rare, serious patterns. For a chatty stream, move to a metric.

For the wider picture of noise reduction on the Davis side, also read reducing Davis alert noise.

Create the alert from a metric extracted from logs (SaaS)

When you want a threshold, extract a metric first. In the pipeline, open Metric extraction, give the metric a name and an ID, then add the same Matching condition. Add dimensions. To link logs to entities, add dt.source_entity or another field holding an entity identifier.

Next, go to Anomaly Detection and create a custom alert in simple mode. The page Set up custom alerts based on metrics extracted from logs describes four steps:

1. set the alert scope;
2. reference the metric in DQL, with by: {dt.source_entity} to link the alert to an entity;
3. choose the conditions and analyzer that generate the Davis event;
4. write the title and description.

Two buttons save you from guessing. Suggest values proposes a threshold. Preview estimates how many alerts would have fired over the last two hours. The documentation does not give the full metric query: check it for the exact syntax before writing it.

Metric analyzers run every minute, so this route is not suited to real-time alerting. The documentation also recommends low-cardinality dimensions.

Cutting the noise on the metric side: window and missing data

The static threshold analyzer in Dynatrace SaaS, described on the Anomaly detection configuration page, does not react to a single sample.

• Sliding window. By default, any three one-minute samples out of five must violate the threshold to raise an event. The window can reach 60 minutes.
• Closing. By default, five one-minute samples back to normal close the event. New violations are added to the existing Problem instead of creating another.
• Missing data. This option, in the advanced properties, is combined with the threshold using OR. Dynatrace recommends disabling it for sparse streams, because it can cause an alert storm.

This ties directly to an SLO: one isolated error log barely touches the error budget, while a run of them inside the window is a real signal. For data delayed by five minutes, the documentation advises a window of at least ten minutes. For the same reasoning on metric events, see creating a Dynatrace alert step by step.

Dynatrace Managed: log metric then metric event

On Dynatrace Managed, the documentation describes the Logs Classic log metrics. The Log metrics (Logs Classic) page in the Managed documentation gives the procedure.

1. Go to Settings > Log Monitoring > Metrics extraction and choose Add log metric.
2. Enter the key after the mandatory log. prefix, the query in Matcher, then a Measure: « Occurrence of log records » (from version 1.206) or « Attribute value » (from 1.229).
3. Add a dimension if needed, then save.
4. Go to Settings > Anomaly detection > Metric events, choose Add metric event and build the event on your metric.

Here noise is tuned with the matcher and cardinality. The documentation sets limits of 1,000,000 dimensions per metric and 50,000,000 across all metrics, and advises narrowing the query first. A new metric holds nothing until new matching logs arrive. Editing the key creates a new metric: history stays under the old key.

The metric event settings (window, missing data) are explained in creating a Dynatrace alert. To limit false positives on Managed, see Dynatrace Managed false positives.

How to check that it works

1. Replay the DQL query in Logs and check that it returns the intended lines, and nothing else.
2. Produce or wait for a matching log, then open Problems: a Problem should appear with the name and description from the template.
3. With a custom alert, use Preview to count the alerts it would have raised over the last two hours.
4. On Managed, query the log. metric in Data Explorer and confirm it carries values before enabling the metric event.

Best practices

• An alert must call for human action. Otherwise, make it a chart.
• Always tie the alert to an entity with dt.source_entity.
• Estimate the volume before enabling, then review the alert count after a week.
• To route notifications, see Dynatrace alerts to Slack and Teams, and to build other metrics, creating a custom metric.

Frequently asked questions

Should I use a metric or an event to alert on logs?

An extracted event is enough for a rare pattern that should open a Problem straight away. A metric fits when you need a threshold, a duration or an analyzer, at the price of an evaluation every minute.

Why does my Dynatrace log alert not fire?

First check that the matcher fits the log as it arrives in the pipeline, and that the route sends the log to the edited pipeline. Routing runs before processing, so a field created later cannot be used there.

How do I avoid bursts of alerts on a chatty log stream?

Tighten the filter, move to a metric, lengthen the sliding window, and disable missing-data alerting for a sparse stream. Later violations are added to the open Problem instead of creating new ones.

Conclusion

A good Dynatrace log alert starts with a precise filter and is tuned by duration, not volume. Choose the event for the rare, the metric for the drift, and keep the DQL query as a last resort. The gain shows in MTTR and in the number of nights nobody is woken for nothing. Reference: Set up alerts based on events extracted from logs.

span.observer is an independent blog, not affiliated with Dynatrace LLC. Dynatrace and Davis are trademarks of Dynatrace LLC. The information comes from public documentation; check it against your version before changing anything in production.

Originally published on span.observer. French version: https://span.observer/fr/alerte-logs-dynatrace-reduire-bruit/.