Building Observability and Monitoring for Modern Applications with Actuator, Prometheus and Grafana

#backend #monitoring #springboot #java

In today’s world of distributed systems and microservices, ensuring our application is observable and monitorable is just as important as building the core functionality. We’ve already set up critical features like a NGINX load balancer, a rate limiter, and a circuit breaker, the next step is to focus on observability and monitoring.

In this blog post, We’ll walk through how to add Spring Boot Actuator, Prometheus, and Grafana to our application to build a robust observability stack. This will help us visualize our application’s health, track performance metrics, and troubleshoot issues quickly and efficiently.

What Is Observability?

Observability refers to your ability to understand the internal state of a system based on the data it produces. The three pillars of observability are:

Metrics: Quantifiable data points (e.g., request rates, memory usage, CPU utilization).
Logs: Record of events (e.g., errors, warnings, or business events).
Traces: Follow a request as it flows through multiple services.

By focusing on metrics and logs, we can build powerful dashboards and alerts that ensure your application remains performant and reliable.

Why Observability Is Important for Our Application

Our current application architecture already has essential components:

NGINX Load Balancer: Distributes traffic across servers.
Rate Limiter: Prevents overloading by limiting the number of requests.
Circuit Breaker: Ensures resilience by stopping calls to failing services.

However, while these tools enhance performance and reliability, they don’t tell us why something might be failing or how our system is performing under load. Observability tools like Actuator, Prometheus, and Grafana will:

Track real-time metrics for application health and performance.
Help visualize trends and potential bottlenecks.
Trigger alerts when metrics cross critical thresholds.

The Observability Stack

Add to your pom.xml file these dependencies:

<dependency>
   <groupId>io.github.resilience4j</groupId>
   <artifactId>resilience4j-micrometer</artifactId>
   <version>2.2.0</version>
</dependency>

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
  <version>1.14.1</version>
</dependency>

Update the configurations of your application.properties

resilience4j.circuitbreaker.metrics.enabled=true

management.health.circuitbreakers.enabled=true
management.endpoints.web.exposure.include=health,metrics,circuitbreakers,prometheus
management.endpoint.health.show-details=always
management.endpoint.health.access=unrestricted
management.endpoint.prometheus.access=unrestricted
management.prometheus.metrics.export.enabled=true

Explanation

management.endpoints.web.exposure.include=health,metrics,circuitbreakers,prometheus
This line is exposing the URI from actuator, so we can consume URIs like:

actuator/
actuator/health,
actuator/metrics,
actuator/circuitbreakers,
actuator/prometheus

Using prometheus with docker

In our docker-compose.yaml file, we create a service for prometheus:


prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    networks:
      - app_network
    volumes:
      - ./prometheus/prometheus.yaml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus

Config file for prometheus

At the root of you project create a folder called prometheusand inside that a file called prometheus.yaml


global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'spring-boot-app'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets:
          - 'spring-server-1:8080'
          - 'spring-server-2:8080'
        labels:
          environment: development
          application: spring-boot

Now, when we run:

docker compose up -d

A prometheus container will start, and consume metrics from the URI actuator/metrics from our spring-boot-servers.

We can see a dashboard at http://localhost:9090/, for example:

But, this is not cool. We want to see some graphs, and for this we use Grafana.

Add Grafana

Update your docker compose file with another service:


grafana:
    image: grafana/grafana:latest
    networks:
      - app_network
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana

Now you can access grafana dashboard on http://localhost:3000

First they will ask your credentials, just write admin for user and password.

Configure Prometheus

On the left up menu, go to connections > add new connection and search for Prometheus

Configure the connection url like this:

Click on the button save & test, if everything is fine you can start choose your dashboard.

Dashboards

Go to Grafana Dashboards and choose a dashboard for you.

For this, I choose the Spring Boot Resilience4j Circuit Breaker (3.x)

If everything works fine you will see something like this:

Feel free to browse other dashboards.

Final Words

By integrating Actuator, Prometheus, and Grafana into our application, we’ve taken a major step toward building a highly observable system. With metrics, logging, and monitoring in place, you’ll be able to: