Observability with SigNoz for Java app

We at Infrasity have curated a tutorial showcasing the real-world capabilities of OpenTelemetry in Java applications.

Agenda

This article will guide you through the comprehensive process of enhancing Java application observability using OpenTelemetry, complemented by the advanced capabilities of SigNoz. We start with the auto-instrumentation of the Java application, simplifying data collection while maintaining performance. We delve into manual instrumentation for scenarios demanding greater control, providing insights into its necessity and implementation. This data is then processed by a local OpenTelemetry collector, a pivotal step in refining and preparing it for comprehensive analysis. The culmination of this process involves sending all collected data to SigNoz, an open-source platform distinguished by its integrated approach to APM, logs, metrics, and more.

The tutorial will showcase the following:

1. Auto-instrumentation of the Java application
2. Sending the data to a local open telemetry collector, which does some sort of data processing
3. Send all data to SigNoz for monitoring and visualization for end users
4. Manual instrumentation, a scenario where it can be needed, and how to proceed.

How does open telemetry help in the observability of your application?

OpenTelemetry enhances application observability by providing standardized tools and libraries for collecting distributed traces, logs, and metrics. It enables developers to instrument code quickly, supports multiple programming languages, and interacts with various observability backends.

OpenTelemtry provides real-time insights into metrics, errors, and transaction flows, helping developers and IT teams quickly identify and address performance issues. They offer code-level visibility, distributed tracing, error tracking, and infrastructure monitoring to optimize application performance and user experience.

Why SigNoz when we have other APM tools?

SigNoz is an open-source Datadog or New Relic alternative.

A single tool for all your observability needs - APM, logs, metrics, exceptions, alerts, and dashboards powered by a powerful query builder. You don’t need to manage multiple tools for traces, metrics, and logs. Get great out-of-the-box charts and a powerful query builder to dig deeper into your data.

Installation & Setup

Before we can see the observability function with SigNoz, we will understand how to integrate OpenTelemetry & Signoz with your application for observability.

Introduction to the tech stack
Here’s the architecture of the sample Java application along with OpenTelemetry and SigNoz.

The below diagram depicts:

• The frontend, query service, and the backend components of the Java application.

• Traces and logs will be sent to the Otel Collector.

• Signoz is leveraging Otel to collect and display the data.

Pre-requisites

• Java 8 or newer
• Maven
• SigNoz

Installing Maven​

Maven is a build automation and project management tool that simplifies the development process by managing dependencies, defining a standardized build lifecycle, enforcing project structure conventions, and providing a centralized repository.

To install Maven, follow the below steps:

$ cd ~
$ mkdir maven
$ cd maven
$ curl -L https://dlcdn.apache.org/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.zip -o maven.zip
$ echo -n '\n export PATH=~/maven/apache-maven-3.8.4/bin:$PATH' >> ~/.zshrc
$ source ~/.zshrc

Verify maven using the below command.

$ mvn -version

Running Java application​

Below are the steps to run the sample Java application with OpenTelemetry:

Clone the sample Spring Boot app
We will be using a sample Java app at this GitHub repo.

$ git clone https://github.com/abbhishekpandey47/demo

Now, how do we send the data to an open telemetry collector, which does some processing for you?

So, let's set this up as well. Very easy!

Setting up an Open Telemetry agent

To instrument Java applications, OpenTelemetry provides a Java JAR agent that can be attached to any Java 8+ application.

Download the latest version of the Java JAR agent, and copy the jar agent file in your application code. You can place the agent jar under the folder named Agents.

Note: You must add this jar to each service if multiple Java applications run as a service.

Placing OpenTelemetry Java Jar agent under a folder named agents

• Let's bind the agent to the application so that it starts collecting the metrics, traces, and logs.
• Create a start.sh script with the below environment variables and move it to the scripts folder.
• Get the script here. Read more about OTEL variables.

JAVA_OPTS="${JAVA_OPTS} \
    -Xms${JAVA_XMS} \
    -Xmx${JAVA_XMX} \
    -Application.name=demo-application \
    -Dotel. traces.exporter=otlp \
    -Dotel.resource.attributes=service.name=demo-service-java \
    -Dotel.service.name=demo-service \
    -Dotel.java agent.debug=false \
    -javaagent:../agents/opentelemetry-javaagent.jar"

As we talk about SigNoz, we will send all the data to SigNoz for monitoring and visualization.

Installing SigNoz

SigNoz can be installed on macOS or Linux computers using a simple install script.

The install script automatically installs Docker Engine on Linux. However, before running the install script, you must install Docker Engine on macOS.

$ git clone -b main https://github.com/SigNoz/signoz.git
$ cd signoz/deploy/
$ ./install.sh

When you are done installing SigNoz, you can access the UI at http://localhost:3301

Note: If you are a new user, you must create an account on SigNoz to start visualizing the application-level metrics.

SigNoz dashboard shows services from a sample app that comes by default when you set up Signoz for the first time.

Setting up SigNoz as the OpenTelemetry backend

To configure OpenTelemetry for collecting and exporting telemetry data, you must define the OTLP (OpenTelemetry Protocol) endpoint. This comprises the IP address of the machine hosting SigNoz and the port number on which SigNoz is listening. The OTLP endpoint for SigNoz is structured as follows: <IP of the machine>:4317.
If you have installed SigNoz on your local machine, then your endpoint is 127.0.0.1:4317

Now, update the OTLP endpoint under the script we created in the previous step.

-Dotel.exporter.otlp.traces.endpoint=http://localhost:4317

The final script would be as follows.

Run your java application

• Now, you need to run your microservices.
• Run demo-service:

$ cd demo
$ mvn clean install -Dmaven.test.skip # Build demo-0.0.1 jar
$ cd scripts
$ sh ./start.sh # Run demo-service with OTEL java agent

• Now, if you go to the SigNoz dashboard, you will notice there is a new app in the list that we have just built.

• You can do some CRUD operations via swagger API documentation: http://localhost:8080/swagger-ui/index.html#/

• Now Click on the demo service application from the list in the SigNoz Dashboard. You can see latency in requests and responses from the demo service app and the operations that took place within the app.

• You can start visualizing DB Call Metrics, latency, and Duration of calls

• It also helps you track any exceptions that have occurred in the app.

• You can see how the applications connect by clicking the Service Map.

• You can view traces of your application.

• View your app logs.

• You can also create Alerts to get notified whenever something has gone wrong.

How about experimenting with the creation of a Trace-based Alert?

We have developed an alert that involves querying our service name and counting the HTTP methods grouped by the specific HTTP method. Subsequently, we established a threshold value to determine when the alert should be triggered and then saved the configuration.

After visiting the API documentation, we repeatedly requested the PUT API. If you check the Triggered Alerts section now, you should see that an alert has been triggered.

Moving to Manual Instrumentation of the application

If you desire to send custom traces or metrics, it's entirely achievable. By leveraging the OpenTelemetry SDK, you can create your own spans tailored to your specific requirements.

You've added the OpenTelemetry SDK dependency to your pom.xml file to facilitate this. This allows you to utilize the SDK to instrument your application and gather custom telemetry data for further analysis and monitoring.

Then, to implement custom traces and metrics, we created a Manual Instrumentation class in the Java application. This class serves to manually instrument your code, providing the flexibility to define and capture specific telemetry data according to your unique needs. Through the OpenTelemetry SDK integration, this manual instrumentation empowers you to contribute custom traces, enhancing the granularity of data collection for more precise analysis and monitoring in your application.

Subsequently, The object of the Manual instrumentation class is created within the Greet Controller Class. The integration is done by invoking the CreateTrace method from that object under the /sayHello API endpoint. This step ensures the custom trace creation logic is executed while processing the /sayHello API at the desired point in your codebase.

We need to restart the application to apply the changes made to the Java application, including the manual instrumentation. Before doing so, it's essential to rebuild the JAR file to incorporate any recent updates.

$ mvn clean install -Dmaven.test.skip

After successfully building the JAR file, restart your Java application. Once the application runs again, access the Swagger UI and hit the /sayHello endpoint to trigger the manual instrumentation logic.

Example 1: Custom Trace Data

Example 2: Custom Trace Data

In conclusion, this tutorial provided an overview of OpenTelemetry, an observability framework that enables the collection, processing, and export of traces and metrics from applications. The focus was on SigNoz, an APM (Application Performance Monitoring) tool, and its integration with OpenTelemetry. The tutorial covered sending data to a local OpenTelemetry collector, which performs initial processing before forwarding the data to SigNoz for monitoring and visualization.

The tutorial also discussed auto-instrumentation, a mechanism by which a Java application can be automatically instrumented for tracing without manual intervention. This automated approach streamlines the process of integrating OpenTelemetry with applications.

Furthermore, the tutorial delved into manual instrumentation, providing a scenario that might be necessary. It explained how to manually instrument a Java application using OpenTelemetry, showcasing the creation of spans and adding custom attributes to capture specific traces. This manual approach allows developers to tailor instrumentation to their particular needs.

By covering auto-instrumentation and manual instrumentation, the tutorial offered a comprehensive understanding of how OpenTelemetry can be seamlessly integrated into Java applications for effective monitoring and observability, with SigNoz as a valuable tool in the APM ecosystem.

To learn more about such awesome content related to DevOps or infrastructure engineering, check out Infrasity.