In addition to the Prometheus: Alertmanager’s alerts receivers and routing based on severity level and tags post post.
Have an Alertmanager config with routes.
The task is – send all alerts from a Dev-environment into a “/dev/null”.
To do this – create an empty receiver:
...
receivers:
- name: 'blackhole'
- name: 'default'
slack_configs:
- send_resolved: true
title_link: 'http://dev.monitor.example.world/prometheus/alerts'
title: '{{ if eq .Status "firing" }}:confused:{{ else }}:dancing_panda:{{ end }} [{{ .Status | toUpper }}] {{ .CommonAnnotations.summary }}'
text: "{{ range .Alerts }}*Priority*: `{{ .Labels.severity | toUpper }}`\nMonitoring host: {{ .Labels.monitor }}\n{{ .Annotations.description }}\n{{ end }}"
...
And in routes add new rules – check the env tag used in our monitoring with the match_re and send all matches to the receiver: blackhole:
...
receiver: 'default'
routes:
# capture All INFO
- match:
severity: info
receiver: default
routes:
# forward Dev INFO to the 'blackhole'
- match_re:
env: .*(-dev).*
receiver: blackhole
# capture All WARN to the 'warning' with P3
- match:
severity: warning
receiver: warning
routes:
# forward Dev WARNING to the 'blackhole'
- match_re:
env: .*(-dev).*
receiver: blackhole
# forward Stage WARN to the 'default'
- match_re:
env: .*(-stage).*
receiver: default
# capture All CRIT to the 'critical' with P1
- match:
severity: critical
receiver: critical
routes:
# forward Dev CRIT to the 'blackhole'
- match_re:
env: .*(-dev).*
receiver: blackhole
# forward Stage CRIT to the 'default'
- match_re:
env: .*(-stage).*
receiver: default
...
Done.
Similar posts
- 03/06/2019 Prometheus: blackbox-exporter probe_http_status_code == 0 and its debug (0)
- 03/10/2019 Prometheus: RTFM blog monitoring set up with Ansible – Grafana, Loki, and promtail (0)
- 08/15/2018 Prometheus: проверка отсутствия метрик – avg_over_time() (0)
- 10/30/2018 Prometheus: Alertmanager’s alerts receivers and routing based on severity level and tags (0)
Top comments (0)