Photo by Ibrahim Boran on Unsplash

I already wrote about my flow when I deploy one container per service and use it for all of my projects. So it's easier to get a development process up to speed.

This time I'll tell you about connecting containerized services so they can talk to each other. And we'll start with somewhat easy things like Prometheus and Grafana. And after that it will be a no-brainer for us to spin up an ELK stack. Let's go! 🚀

Side Note

All code examples and configs are available on my GitHub repo.

Metrics

If you didn't know what Prometheus is used for - it collects various metrics from a service. And a little side thing we need to do before setting up our God of Fire is to create a service we'll be monitoring. I chose it to be a simple Flask app because it requires almost zero efforts.

Flask App

Create a flask_app directory and put next two files there:

# flask_app/app.py
from flask import Flask
from prometheus_flask_exporter import PrometheusMetrics

app = Flask(__name__)
metrics = PrometheusMetrics(app)

# flask_app/wsgi.py
from app import app

Also create requirements.txt right next to previous two:

click==7.1.2
Flask==1.1.2
itsdangerous==1.1.0
Jinja2==2.11.3
MarkupSafe==1.1.1
prometheus-client==0.9.0
prometheus-flask-exporter==0.18.1
Werkzeug==1.0.1

Then create Dockerfile, same place:

FROM python:3.9

ENV PYTHONUNBUFFERED 1
ENV PYTHONDONTWRITEBYTECODE 1

EXPOSE 8000

RUN apt-get update \
  && apt-get install -y build-essential \
  && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
  && apt-get clean -y && rm -rf /var/lib/apt/lists/* \
  && pip install --upgrade pip

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

ENTRYPOINT ["gunicorn", "wsgi:app", "--bind", "0.0.0.0:8000"]

That's quite mouthful but all it does is:

• creating an image from python 3.9;
• opens up port 8000;
• updates everything;
• and installs our great app.

Now we build!

docker build . -t flask_app

And now we run!!

docker run -d -p 8000:8000 flask_app --name flask_app

And on http://0.0.0.0:8000/metrics we can see a bunch of stuff!!1 💪

Prometheus

It's time to start metering. Open my our favorite tool to manage containers - Portainer. And first, create a volume. Actually, two volumes. If you've read my previous article you know how it goes. But in case you didn't (why didn't you tho?) - go to "Volumes" tab and hit that blue "Add volume" button. First volume will be used for Prometheus data. So let's name it prometheus_data. Second will be used for Prometheus config, therefore the name prometheus_config_data.

"Why data for config?" you might ask. Well, because apparently you can't access the Prometheus container via the Portainer console button. Or I'm just stupid 🤷‍♂️ But we need a way to edit config files without rebuilding container every time. Thus volume for config files.

Creating container, finally. Go to "Containers" and smash the "Add container" button. Then look at the screenshot below and fill values for Name, Image, Ports and Volumes. Maybe change Restart Policy to Unless stopped.

Then hit the "Deploy the container" button and wait a little.

Configure Prometheus

Since we don't know how to can't connect to a Prometheus container we will edit it's config file via terminal session. If you're using nano like I do then just execute next line and make it look the same as below:

sudo nano /var/lib/docker/volumes/prometheus_config_data/_data/prometheus.yml

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'flask_app'
    metrics_path: '/metrics'
    static_configs:
    - targets: ['172.17.0.1:8000']

It's the same default config but with additional job_name at the end. I decided to be explicit with metrics_path here, showing that we can change that if we need to. But the most important part is targets! You can just believe me that on Linux your host machine's IP address will be 172.17.0.1 for your docker containers. Or you can visit a "Network" tab in Portainer and take a look at "IPV4 IPAM Gateway" for a bridge interface.

Now restart Prometheus container and visit http://0.0.0.0:9090/targets. You should see "State" UP for both targets.

If we refresh our Great Flask App that serves us 404 on http://0.0.0.0:8000/ and then go to graph for total requests, we should see.. well.. a graph for total requests.

Cool, eh? Let's make it beautiful! 🐹

Grafana

Again, create a volume, for example grafana_data. Then create container as on screenshot below

Hit deploy, wait and go to http://0.0.0.0:3000/. Default login/password is admin/admin. You can change that after login.

Configure data source on http://0.0.0.0:3000/datasources. Or locate "Data Sources" tab hovering on ⚙️ sign on the left. Then click the blue "Add data source" button, select "Prometheus". Very important, as before, "URL" parameter inside the "HTTP" section should be 172.17.0.1:9090. Because Grafana can't access Prometheus container by host name. So we tell it (her?) to connect to our host machine IP address on the port where Prometheus is running. Then smash "Save & Test". As always, you can try to set "URL" to prometheus_container_name:port and after saving observe "HTTP Error Bad Gateway".

Now we can hover over ➕ sign and click on "Dashboard" under "Create". Or just visit http://0.0.0.0:3000/dashboard/new. Then smash "Add new panel". On the right side change "Title" to whatever your heart desires. I chose simple "Flask Total Requests". Below graph set "Metrics" to flask_http_request_total. Here is a handy screenshot for you:

Then click "Apply" in the top right corner. And now you have Grafana which makes beautiful graphs from Prometheus data which monitors your application!

In the next article we will set up ELK. But I guess you already know how to do that 😉