Complete Guide: How to use Grafana with a custom Node API.

Data visualization is important for harnessing the value in the data we have at our disposal. Grafana (described as The open observability platform) is used by thousands of companies to monitor everything. It makes data visualization and monitoring simpler.

Grafana basics

Grafana can be downloaded in various ways from their site.

Grafana is available for Linux, Windows, Mac, Docker, and ARM.
The download page details how it can be downloaded and installed for each one of these options.

Once Grafana is installed and running, you'll need to create a dashboard and at least one panel. A panel is the part of the dashboard that will have a specific visualization.

When you add a new panel in the latest version of Grafana (7.1.4 at the time of writing this article), a visualization of random walk data over time will be automatically created for you.

Data

In the panel edit screen, you can see the visualization and three tabs at the bottom; namely, the Query, Transform, and Alert tabs. The Query tab has a dropdown with options of data sources you've added to your project as shown in the image below.

You need to create your own data source. To do this, head back to the main screen, hover over the settings cog, then click on 'Data Sources.'

The dependency (JSON)

We'll need a plugin to help us fetch and format our data for Grafana. This example makes use of the JSON Datasource Plugin. You need to download and install the plugin as it's shown on the site.

Adding the data source

Once you've got the plugin installed, click on add data source then search for JSON plugin under Add data source. Once you have it, click select.

The most important field on this screen is the URL. In here, populate your custom Node.JS endpoint. The name field is simply for you to be able to tell between your different data sources so you can name it anything you like as I did below 😂.
.

The Restful API

The plugin's documentation stipulates that you need to implement at least 4 endpoints. I'll walk you through what these actually do and when they are triggered.

As mentioned in the docs, you need a GET/ endpoint that returns a status code: 200 response. This is used to test whether your API is running. Without the expected response, Grafana won't add your data source and will pop up a "HTTP ERROR Bad Gateway" message.

Click on the 'Save and Test' button at the bottom of the screen to save your new data source.

Head back to your panel and click on edit. Now, when you click on the Query dropdown, your aptly named data source should appear as the first alternative option in the list.

Under the 'A' query, there are a few things to discuss.

The first field labeled 'Format as' controls the kind of data that is sent to your backend with the data request. It has two options, Time and Table. You can decide to perform different actions on your backend based on this. For this example, we won't care about the field.

The next field, labeled Metric, is important to us. To populate this field, the data source will make a POST request to the endpoint you specified and suffix it with a 'sub endpoint': "/search". so in our example, to populate this dropdown, the JSON plugin will make a POST request to localhost:4400/myEdnpoint/search. This means that your server should make a 'search' endpoint available.

In my Node.JS + Express restful API, this is what the example code would look like:

module.exports = router.get(
  '/myEndpoint',
  handle_errors(async (req, res) => {
    res.status(200).send('successfully tested');
  })
);

module.exports = router.post(
  '/myEndpoint/search',
  handle_errors(async (req, res) => {      
    let data = [ { "text": "upper_25", "value": 1}, { "text": "upper_75", "value": 2} ];
    res.status(200).send(data);
  })
);

The first endpoint GET/ simply returns a status code 200 response.
The second endpoint with '/search' is a POST and returns a key-value part of text and value. The value will be used to query for data we want to visualize.

Once you click on the Metric field of your choice as presented in the dropdowns, the plugin will make a POST request to a 'sub endpoint' with '/query'. So in our example, once a choice is made from the dropdown, the JSON plugin will make a POST request to localhost:4400/myEdnpoint/query with added information passed to it.

This means our restful API needs to expose this endpoint. Our example implementation is shown below:

module.exports = router.post(
  '/myEndpoint/search',
  handle_errors(async (req, res) => {      
    let data = [ { "text": "upper_25", "value": 1}, { "text": "upper_75", "value": 2} ];
    res.status(200).send(data);
  })
);

As shown in the image of my debug window belong, the plugin makes a POST request and passes a lot of data in the body.

The body object has a field called targets which is an array of added user-input information. The first element of this array provides information from our first query in Grafana. This includes information about whether we want data formatted as Time series or a Table, the data source name, any additional user data, and importantly, the 'target' which is the value of the selected metric on the Grafana UI. I clicked on the 'upper_75' option and as dictated by our response in the /search/ endpoint, the value of this field is 2. This is also visible in the debug window shown above.

On the Grafana query editor window, we also have a field where we can provide additional data in JSON form as shown in this image

This data will be sent to the /query endpoint with the POST request once a metric is chosen.

This data will also be in the target array's first element under the 'data' object. This can be seen in my debug window below.

Data response.

Now that we have the required endpoints to accept requests for status checking (GET/ 200), options (/search), and actual data retrieval (/query), we need to understand how to format our response for Grafana to be able to visualize our data.

In the screenshots above, data is retrieved from a function called getQueryData(). All this function does is return an array with data formatted for visualization by Grafana as shown below.

const getQueryData = () => {
  return [
    {
      "target":"pps in",
      "datapoints":[
        [622,1450754160000],
        [365,1450754220000]
      ]
    },
    {
      "target":"pps out",
      "datapoints":[
        [861,1450754160000],
        [767,1450754220000]
      ]
    },
    {
      "target":"errors out",
      "datapoints":[
        [861,1450754160000],
        [767,1450754220000]
      ]
    },
    {
      "target":"errors in",
      "datapoints":[
        [861,1450754160000],
        [767,1450754220000]
      ]
    }
  ]
};

Let's understand this response. This is an array of 4 objects. These objects are labeled 'target'. This should also tell us that the target field is likely to be a key of sorts. Since we'll be plotting a simple bar graph, the target field will be labeling our x-axis values. The next field in the object is datapoints which has to be a 2-dimensional array.

As shown in the Grafana documentation, the datapoints property is of type TimeSeriesPoints.

...which is of type [][] (2D Array) as shown below.

You can think of the 2D Array as a Table with values and the time. The first entry's datapoints are represented in the table below. The time is shown as a Unix timestamp in milliseconds.

The Grafana visualization is taking shape but it isn't the bar graph we want yet. .

At this point, we've done everything necessary on our Node backend. The rest is up to the front-end configuration of Grafana.

Click on the 'show options' button on the top right corner of the edit screen. This brings up a panel with configurations for your display panel.

Under "Visualization," you can see the different types of visualizations you can create on Grafana. We'll go with the bar graph.

We need to change from being a time-based visualization to one where our 'target' is the independent variable. We need to change the x-axis. Change the x-axis mode from Time to 'Series' and voila; we now have a bar graph. Play around with the rest of the configurations to see how they change your visualization.

Grafana is a powerful tool in the right hands. I hope this was useful to you. Keep learning.