This post originally published at QAInsights Blog.
You can check out my videos about LoadRunner integration with Grafana in my YouTube channel. Recently I have started working extensively on JMeter, primarily in HTTP, MQTT, and JDBC Sampler and will be working on NeoLoad pretty soon. I got a challenge in my projects where developers would like to monitor the critical metrics when the performance test is on. I started sharing the Dynatrace dashboard link, but for non-technical members Dynatrace information is overwhelming. Our stack has Prometheus - time series database with Grafana in the front end. Then I started exploring about the integration and successfully implemented JMeter > Prometheus > Grafana integration and shared the link with my team. It is well-received. In this blog post, we will see about JMeter, Prometheus, and Grafana Integration.
Little Background
In JMeter Plugins repository, there are backend listeners available for Kafka, ElasticSearch, and Azure. By default, JMeter supports Graphite and InfluxDB. But there is no backend listener available for Prometheus.
Fortunately, I found this excellent open source project to export the JMeter metrics for Prometheus. Thanks to Jeff Ohrstrom.
Now we shall see high-level diagram about the integration.
As you observed in the above diagram, JMeter + Prometheus plugin exports the JMeter metrics to Prometheus database. If you want to see the visualization, you need to configure the data source in Grafana.
The main intention of the plugin is to export the results to database (Prometheus).
What is Prometheus?
Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud.
What is Grafana?
Grafana is the open source analytics and monitoring solution for every database.
What is JMeter-Prometheus Plugin?
It is an open source project which helps to export the JMeter metrics to Prometheus.
Prerequisites
- Apache JMeter
- JMeter-Prometheus Plugin
- Prometheus
- Grafana
Recommended versions for above components is latest and stable release. I am in Windows 10 Home Edition.
Installing JMeter-Prometheus Plugin
There are multiple ways to place the jar file for JMeter. Very simple method is to download the jar file from the latest releases in GitHub and place it in the <JMETER_HOME>\lib\ext.
Download JMeter-Prometheus Plugin from here
Validate the installation by creating a simple script in JMeter, and then navigate to Right Click on Thread Group > Add > Listener > Prometheus Listener
Or Right Click on Thread Group > Add > Config Element > Prometheus Metrics
Install Prometheus
It is very straight forward to install Prometheus. Download Prometheus for your OS and extract the contents and keep it ready. In few moments, we will fire up Prometheus.
Install Grafana
Like Prometheus, it is easy to install Grafana. Head to this download section and install it. By default, it uses the port 3000.
Configuring JMeter-Prometheus Plugin in JMeter
After designing your test, you need to configure JMeter-Prometheus plugin in your test plan. Detailed instruction has already available in the GitHub page. Please read it twice for better understanding. I am not going to reiterate again in here :)
To demonstrate, I have configured the listener as shown below.
Configuring Prometheus
Now, it is time to configure Prometheus so that JMeter-Prometheus plugin will export the results to Prometheus database.
Go to Prometheus installation folder and open prometheus.yml in your favorite editor.
Add the below job details to your yml file. Below configuration helps to scrape the metrics from the port 9270.
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']
- job_name: 'jmeter'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9270']
Launch Prometheus
Once the configuration is completed, launch command prompt and change the directory to the prometheus installation folder and fire up the below command.
prometheus.exe --config.file=prometheus.yml
Now launch, Prometheus by launching http://localhost:9090/graph as shown below.
Launch JMeter Test
We are good with Prometheus. Now it is time to validate the JMeter test plan. Configure your thread group with couple threads and loop infinite and hit run.
Launch, http://localhost:9270/metrics as shown below. If you are seeing the below metrics, everything is working fine. If not, please let me know the issue you are facing.
Viewing JMeter metrics in Prometheus
As highlighted in the above metrics snapshot, we will see how to view the metrics in Prometheus. Just copy jvm_classes_loaded
and go to Prometheus and paste it in the query text box and hit enter. You can see the value as shown below.
Click on Graph tab to view the visualization. You can customize how you want to visualize. Below is the graph which displays 95 percentile response time of each transaction.
Where is Grafana :) ?
If you do not want to integrate Grafana, you can stop here. But Grafana is more powerful than the Prometheus visualization features.
Now, let us configure Grafana to fetch the metrics from Prometheus and create beautiful visualization.
Launch Grafana by navigating to http://localhost:3000, you may need to change the password if you are logging in for first time.
Hover your mouse over the Settings icon and click on Data Sources as shown below.
Click on Add data source as shown below.
Fill up the details as shown below and hit Save & Test
Let us validate the Prometheus data source in Grafana
Hover your mouse over Explore icon and click on it.
Enter jmeter_threads{}
and hit enter the query text box. If the configuration is correct, you will see the graph. Note that your JMeter test is still running. If stopped, you need to change the time picker value at top right corner.
Creating Grafana Dashboard
Hover your mouse over plus icon and click on Dashboard.
Click on Add Query button.
Change the Query dropdown to Prometheus as shown below.
Enter the query jmeter_summary{quantile="0.95"}
and hit enter. You can customize the graph as desired.
Here is my simple dashboard.
Sample Grafana Dashboard
https://github.com/johrstrom/jmeter-prometheus-plugin/tree/master/docs/examples
Adding an excerpt from one of the author.
Another point of the plugin is that it does not need to send metrics to the JMeter controller to aggregate results as this is done by Prometheus itself.
Conclusion
Now you have successfully integrated JMeter with Prometheus and Grafana. If you are satisfied with Prometheus visualization, no need to configure Grafana. But Grafana has more powerful visualization features than Prometheus. You need to setup JMeter-Prometheus plugin where ever your are running JMeter and streamline the data flow into centralized Prometheus database. This will help you to monitor the runtime metrics.
You need to carefully craft the query in Grafana, otherwise you will end up in viewing false metrics.
Please let me know in the comments, if you face any issues. Happy to help you out :)
Top comments (26)
How to integrate Jmeter with promotheus+Graphana and my main requirement is how to fetch the JMeter metrics using external IP address because here i am able to see the JMeter metrics using local host 9270 port but i am not able to see the metrics using IP address of that VMmachine (linux),Could you please help me out with your suggestions
I think you need to open the port in your VM, so that Prometheus can scrape. Try to configure the IP as 0.0.0.0 in your JMeter properties prometheus.ip=0.0.0.0
Hi Naveen,
Thank you for your quick reply, Apologies for the delay.
I configured the prometheus.ip = 0.0.0.0 in jmeter properties file in VM machine, even though I am facing the connection refused error. Please find the attached screenshot FYR.
Could you please help me out.
are you able to access the Prometheus URL ? yourip:9270/metrics
No, i am not able to access the Prometheus URL with
ip Address (curl -q -6 -g http://[240b:c0e0:101:54e2::5b]:9270/metrics) but when when i am trying with local host in (curl -q -6 -g localhost:9270/metrics) it is working as expected
dev-to-uploads.s3.amazonaws.com/i/...
Please find the error screen shot
Why there is a square bracket in your URL?
Where are those properties stored? It should be in the user.properties.
with out square brace's i am facing this error
curl -q -6 -g 240b:c0e0:101:54e2::5b:9270/metrics
curl: (3) IPv6 numerical address used in URL without brackets error
[2]+ Exit 1
Not sure about the property remote hosts. Probably you can remove and try only with the Prometheus IP.
as per your comment i removed the remote hosts and tried even though facing the connection refused error
dev-to-uploads.s3.amazonaws.com/i/...
Where are those properties stored? It should be in the user.properties. ---- it means????
can you please let me know if any other way to sort out this issue
Thanks
I am having the same issue, when I try to use external IP? Do you have any pointers for me?
Sorry, I do not have a Docker environment to reproduce the issue. Can you please raise the issue in GitHub?
I solved this issue. I can provide you details if you want
Great, please do share, let me add it to my blog and quote you :)
Sure I will share the solution over the weekend
Many thanks :)
Add the prometheus.ip=0.0.0.0 in user.properties file of JMeter at apache-jmeter/bin
If you want to run Prometheus on a different machine, use the attached Prometheus_jmeter.yml in the VM(assume VM ip address is XX.XX.XX.XXX) to run the Prometheus.
Run below docker command to get the Prometheus image from docker hub and volume the Prometheus_jmeter.yml file in VM to Prometheus container.
-On local machine, run the JMeter file which has Prometheus listener in it.
Do the Port Forwarding from your local machine JMeter port to remote machine using below command
ssh -R 9270:localhost:9270 usernameofVM@XX.XX.XX.XXX
In your remote machine (VM) run the below command to verify that the port forwarding is working. You can to see all the Prometheus metrics show up in remote machine.
curl -v localhost:9270
You can checks all the targets by using this XX.XX.XX.XXX:9090/targets
Yml file:
scrape_configs:
If you setup the Prometheus server at one place. You can run the Jmeter test from any where if you use the Remote port forward and with adding the "--net=host" while getting the prometheus docker image.
Let me know if you have any questions. you can reach out to me for more details at santoshi.nitya@gmail.com
HI, It's very good information.
I have configured as per the steps that you mentioned above. Here are couple of issues.
1) Metrics are showing in 9090 port instead of 9270 port.
2) After running my jmeter script i don't see the result in metrics.
Can you please help me on this?
Are you using the latest version of Prometheus plugin?
hi Naveen,
I am following this thread to integrate my jmeter test result with prometheus. I did all the setups as suggested in this thread but i am not able to see the data in localhost:9270
I placed the jar file under jmeter and then added the yaml file as suggested in my local n started prometheus in my docker.
But i am unable to get any input in the jmeter url.Attaching the image.Could you please help me.My prometheus url is up localhost:9090/ but i am not seeing any jmeter parameters there to execute.
Please check the log from prometheus to debug.
HI Naveen - I have a very basic setup of jmeter where i have 1 master machine and 4 slave machines, I tried running jmeter on master machine with prometheus listener, it working fine not issue able to see metrics both in http://:9270/metrics and in Grafana as well, but when I started running distrubuted testing (tried to run a test on all 4 slave machines) that is when I am noticing issues the test is running fine but unable load http://:9270/metrics or unable to see metrics. I am not using docker, its a very basic setup and 5 windows machines 1 master and 4 slave machine. I tried adding prometheus portand ip address in jmeter properties of slave machines but no luck.
Great information; thank you!
I really like the idea of pulling server resource metrics with Prometheus.
I want to run JMeter in headless mode for better performance. The JMeter people all say that it is bad to run JMeter with GUI components in a live load test.
How do you recommend using this component when running in headless mode?
It would seem that the right way is to just view the data in Grafana.
Thanks again.
Thanks. You can launch your test using
jmeter -n -t <test.jmx> .....
for headless mode.