DEV Community

Mario García
Mario García

Posted on

Getting Started with Prometheus Alerting

Continuing with articles about monitoring with Prometheus, this time alerts will be configured to know when a server or app stopped by using Alertmanager. Through this article, you will learn how to set up notifications for a MySQL server.

Running MySQL using Docker

First, create a network:

$ docker network create prom-network
Enter fullscreen mode Exit fullscreen mode

Run the following command to get MySQL up and running in a container:

$ docker run -d -p 3306:3306 --name mysql -e MYSQL_ROOT_PASSWORD=root --network prom-network mysql:8.3.0
Enter fullscreen mode Exit fullscreen mode

Running MySQL Server Exporter with Docker

From the documentation of Prometheus, there are a number of libraries and servers which help in exporting existing metrics from third-party systems as Prometheus metrics.

Some of these exporters are maintained as part of the official Prometheus GitHub organization, those are marked as official, others are externally contributed and maintained.

For MySQL, you can use the official MySQL Server Exporter that supports:

  • MySQL >= 5.6.
  • MariaDB >= 10.3

To install it you can follow the instructions in the GitHub repository and you can run it in a Docker container as well.

First, grant permissions to the exporter to access the database. Login to your MySQL server and run the following queries:

$ mysql -u root -p -h
Enter fullscreen mode Exit fullscreen mode
Enter fullscreen mode Exit fullscreen mode

Then, create the configuration file (, adding login details as follows:

user = exporter
password = password
host = mysql
Enter fullscreen mode Exit fullscreen mode

And finally, initialize the container:

$ docker run -p 9104:9104 --name exporter --network prom-network -v $(pwd)/ prom/mysqld-exporter:main
Enter fullscreen mode Exit fullscreen mode

Running Alertmanager using Docker

First, create the configuration file (alertmanager.yml), this file will contain details about how to send notifications and will look as follows:

  receiver: 'mail'
  repeat_interval: 4h
  group_by: [ alertname ]

  - name: 'mail'
      - smarthost: ''
        auth_username: '<your-email>'
        auth_password: "<your-password>"
        from: '<your-email>'
        to: '<some-email>'
Enter fullscreen mode Exit fullscreen mode

Replace <your-email> and <your-password> with your login details. If you have 2FA activated, generate an app password. And replace <some-eamail> with the email where you want to get notifications.

The configuration file above specifies that a Gmail account will be used to send alerts. Check the Alerting section in the documentation of Prometheus for more information.

Then, run the container:

$ docker run --name alertmanager -d -p 9093:9093 --network prom-network -v ./alertmanager.yml:/etc/alertmanager/config.yml --config.file=/etc/alertmanager/config.yml
Enter fullscreen mode Exit fullscreen mode

Running Prometheus in a container

After configuring the exporter and Alertmanager, run Prometheus in a container, but first, create the configuration file (prometheus.yml). You can start with the basic configuration example from the documentation and add the configuration needed for the exporter as described in the GitHub repository.

  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
    monitor: 'codelab-monitor'

  - "rules.yml"
    - static_configs:
        - targets: ["alertmanager:9093"]

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

      - targets: ['localhost:9090']

  - job_name: 'mysql'

      auth_module: [client]

    scrape_interval: 5s

      - targets: ['mysql:3306']

      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
          # The mysqld_exporter host:port
        replacement: mysql-exporter:9104
Enter fullscreen mode Exit fullscreen mode

This block in the configuration file refers to the Alertmanager.

  - "rules.yml"
    - static_configs:
        - targets: ["alertmanager:9093"]
Enter fullscreen mode Exit fullscreen mode

The rules.yml file contains the set of rules that tell Prometheus when to send a notification. The file will look as follows:

  - name: MysqldExporter
      - alert: MysqlDown
        expr: mysql_up == 0
        for: 0m
          severity: critical
          summary: MySQL down (instance {{ $labels.instance }})
          description: "MySQL instance is down on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
Enter fullscreen mode Exit fullscreen mode

It will create an alert whenever the MySQL server is down and once the alert is created, it will send the notification to the Gmail account previously configured.

Check the Awesome Prometheus alerts website for more examples on how to set the rules.

After both configuration files are created, just run the container.

docker run -p 9090:9090 --network prom-network --name prometheus -v ./prometheus.yml:/etc/prometheus/prometheus.yml -v ./rules.yml:/etc/prometheus/rules.yml prom/prometheus
Enter fullscreen mode Exit fullscreen mode

Now Prometheus is up and running, with Alertmanager configured. You can try it by stopping the MySQL server, an alert will be created and an email similar to the following will be sent to the specified email account.

Alert Notification

Support me on Buy Me A Coffee

Top comments (2)

michaelbhall profile image
Michael Hall

Does this awesome a running smtp server on the host to be able to send the email?

poboisvert profile image