DEV Community

Yokogawa-BrigitteZacharia for Yokogawa Technologies Solutions India

Posted on • Updated on

Using Python to store collected data in InfluxDB and visualizing data in Grafana

Introduction

In the previous article, we learnt how to collect data with Modbus TCP communication using Python. This is the 2nd article of the 3-part series, where we explore how to store collected data in InfluxDB and visualize the data in Grafana.

Supported devices

The steps in this demonstration can be performed on the following devices.

  • e-RT3 Plus F3RP70-2L1 (Ubuntu 18.04 32-bit)
  • Raspberry Pi 4 Model B (Ubuntu Server 20.04 32-bit)

The armhf architecture packages run on these devices.

System positioning

The following figure shows the setup required to collect, store, and visualize data on the Edge device over the Modbus TCP communication.

SystemPositioning_ModbusGrafana

The flow involves four events:

  1. Data collection

    Data is collected from an external device by using Modbus TCP communication.

  2. Data storage

    The collected data is stored in Influx DB on the Edge device.

  3. Visualization

    The data stored in InfluxDB is visualized using Grafana.

  4. Monitoring

    The results are monitored in a computer by accessing Grafana in the Edge device.

Prerequisites

To achieve the objective of this article, you must install Grafana and InfluxDB on the Edge device.

Installing Grafana

Follow these steps to install Grafana on the Edge device.

  1. Install the required packages.

    sudo apt install -y apt-transport-https
    sudo apt install -y software-properties-common wget
    wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
    
  2. Add repositories and install Grafana.

    echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
    sudo apt update
    sudo apt install grafana
    

    For more information on installing Grafana, refer to Install on Debian/Ubuntu | Grafana Labs.

  3. Start the service and verify the status.

    sudo systemctl daemon-reload
    sudo systemctl start grafana-server
    sudo systemctl status grafana-server
    

    If the program is executed successfully, the following output is displayed.

    * grafana-server.service - Grafana instance
    Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; disabled; ven
    Active: active (running) since Fri 2021-12-24 07:38:20 UTC; 14s ago
    Docs: http://docs.grafana.org
    Main PID: 2756 (grafana-server)
    Tasks: 12 (limit: 2366)
    CGroup: /system.slice/grafana-server.service
    `-2756 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --p
    Dec 24 07:38:32 ubuntu grafana-server[2756]: t=2021-12-24T07:38:32+0000 lvl=info
    Dec 24 07:38:32 ubuntu grafana-server[2756]: t=2021-12-24T07:38:32+0000 lvl=info
    

Note: If the Edge device is in a proxy environment, configuring proxy settings is mandatory.

Installing InfluxDB

Follow these steps to install InfluxDB on an Edge device.

  1. Install the package that is used to operate InfluxDB from Python.

    python3 -m pip install influxdb
    
  2. Add repositories and install InfluxDB.

    curl -s https://repos.influxdata.com/influxdb.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/influxdb.gpg
    export DISTRIB_ID=$(lsb_release -si); export DISTRIB_CODENAME=$(lsb_release -sc)
    echo "deb [signed-by=/etc/apt/trusted.gpg.d/influxdb.gpg] https://repos.influxdata.com/${DISTRIB_ID,,} ${DISTRIB_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
    sudo apt update
    sudo apt install influxdb
    
  3. Start the service and verify the status.

    sudo systemctl unmask influxdb.service
    sudo systemctl start influxdb
    sudo systemctl status influxdb
    

    If the program is executed successfully, the following output is displayed.

    * influxdb.service - InfluxDB is an open-source, distributed, time series databa
    Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset:
    Active: active (running) since Fri 2021-12-24 07:51:02 UTC; 6s ago
    Docs: https://docs.influxdata.com/influxdb/Process: 4191 ExecStart=/usr/lib/influxdb/scripts/influxd-systemd-start.sh (co
    Main PID: 4192 (influxd)
    Tasks: 9 (limit: 2366)
    CGroup: /system.slice/influxdb.service `-4192 /usr/bin/influxd -config /etc/influxdb/influxdb.conf`
    Dec 24 07:51:01 ubuntu influxd-systemd-start.sh[4191]: ts=2021-12-24T07:51:01.75
    Dec 24 07:51:01 ubuntu influxd-systemd-start.sh[4191]: ts=2021-12-24T07:51:01.75
    

For more information about installing InfluxDB, refer to the official documentation.

Data Collection

Collect data from the device with a Python program that uses PyModbus. In this example, start the Modbus server on the computer instead of the device.

Note: For more details about data collection using PyModubs, refer to the previous article.

Configuring Modbus server

Follow these steps to configure Modbus server:

  1. Start the Modbus server on the computer.
  2. To rewrite the values periodically, modify the program updating_server.py (You can access the license here).

    In this example, we store two time-series data values in the holding register and Boolean data in the coil.

    1. Modify the following code in the updating_write function.

      import random
      def updating_writer(a):
      context = a[0]
      slave_id = 0x00
      
      # Write to coil 1
      fx = 1
      address = 0x01
      values = [bool(random.getrandbits(1))]
      log.debug("coil1: " + str(bool(random.getrandbits(1))))
      context[slave_id].setValues(fx, address, values)
      
      # Writing to holding register 2
      fx = 3
      address = 0x02
      values = context[slave_id].getValues(fx, address, count=1)
      values[0] += random.randint(-10, 10)
      if values[0] > 100:
      values[0] = 100
      elif values[0] < 0:
      values[0] = 0
      log.debug("holding2: " + str(values[0]))
      context[slave_id].setValues(fx, address, values)
      
      # Writing to holding register 3
      fx = 3
      address = 0x03
      values = context[slave_id].getValues(fx, address, count=1)
      values[0] += random.randint(-5, 5)
      if values[0] > 50:
      values[0] = 50
      elif values[0] < 0:
      values[0] = 0
      log.debug("holding3: " + str(values[0]))
      context[slave_id].setValues(fx, address, values)
      
    2. Rewrite the following code at the end of the updating_server.py program.

      # Before change
      StartTcpServer(context, identity=identity, address=("localhost", 5020))
      # After change
      StartTcpServer(context, identity=identity, address=("", 5020))
      
  3. Install the following package and start the program.

    python -m pip install twisted pymodbus
    python updating_server.py
    

Data collection in Edge device

Follow these steps to collect data in the Edge device:

  1. Run the following client.py program in the Edge device to collect the server data every 5 seconds.

    Ensure that you replace the with the IP address of the computer on which the Modbus server is running.

    client.py
    
    #!/usr/bin/env python
    from pymodbus.client.sync import ModbusTcpClient as ModbusClient
    import time
    
    # Connect to Modbus server
    mbclient = ModbusClient('<IP_ADDRESS_OF_SERVER_PC>', port=5020)
    mbclient.connect()
    while True:
    try:
        # Read value of coil 1
        rr = mbclient.read_coils(1, 1)
        co1 = rr.bits[0]
    
        # Read values from holding registers 2 and 3
        rr = mbclient.read_holding_registers(2, 2)
        hr2 = rr.registers[0]
        hr3 = rr.registers[1]
    
        print("coil1: {0}, holding2: {1}, holding3: {2}".format(co1, hr2, hr3))
    
        #Sleep for 5 seconds
        time.sleep(5)
    
    except KeyboardInterrupt:
        # disconnect
        mbclient.close()
        break
    
    except Exception as e:
        print(e)
    

    If the program is executed successfully, the following output is displayed.

    username@ubuntu:~$ python3 client.py
    coil1: False, holding2: 89, holding3: 12
    coil1: True, holding2: 85, holding3: 13
    coil1: True, holding2: 78, holding3: 14
    ...
    # Close with Ctrl+C
    

Note: Although this program enables easier collection of periodic data, the period in which the data is collected is not accurate. If you want to collect data in more accurate periods, check other methods of data collection.

Data storage

Store the data that is collected from Edge device in InfluxDB.

Configuring the database

Follow these steps to configure the database:

  1. Start InfluxDB

    sudo systemctl start influxdb
    
  2. Create a database with the name as sampleDB.

    username@ubuntu:~$ influx
    Connected to http://localhost:8086 version 1.8.10
    InfluxDB shell version: 1.8.10
    > CREATE DATABASE sampleDB
    > exit
    

Storage in database

Follow these steps to store the collected data in the database that you created.

  1. Refer to the InfluxDB sample program and add the code to store data to the client.py program that was created in the step Data collection in Edge device.

    After adding, the code is updated as follows.

    client.py
    
    #!/usr/bin/env python
    from pymodbus.client.sync import ModbusTcpClient as ModbusClient
    from influxdb import InfluxDBClient
    from datetime import datetime
    import time
    
    # Connect to Modbus server
    mbclient = ModbusClient('<IP_ADDRESS_OF_SERVER_PC>', port=5020)
    mbclient.connect()
    
    # Connect to database
    dbclient = InfluxDBClient(host='localhost', port=8086, database='sampleDB')
    while True:
    try:
        # Read value from coil 1
        rr = mbclient.read_coils(1, 1)
        co1 = rr.bits[0]
    
        # Read values from holding registers 2 and 3
        rr = mbclient.read_holding_registers(2, 2)
        hr2 = rr.registers[0]
        hr3 = rr.registers[1]
    
        print("coil1: {0}, holding2: {1}, holding3: {2}".format(co1, hr2, hr3))
    
        # Write to database
        json_body = [
            {
                "measurement": "sample_measurement",
                "time": datetime.utcnow(),
                "fields": {
                    "coil_1": co1,
                    "holding_register_2": hr2,
                    "holding_register_3": hr3
                    }
            }
        ]
    
        print("Write points: {0}".format(json_body))
        dbclient.write_points(json_body)
    
        # Skip for 5 seconds
        time.sleep(5)
        except KeyboardInterrupt:
    
        #Disconnect
        mbclient.close()
        dbclient.close()
        break
    
        except Exception as e:
        print(e)
    

    Note: Data is written to the SD card everytime data is stored in the database. If you want to reduce the frequency with which data is written to the SD card, use an appropriate storage medium such as RAM disk or network disk, and specify them as storage destinations in the InfluxDB settings.

  2. Start the program and wait for a few seconds.

    python3 client.py
    # Close with Ctrl+C
    
  3. Run the following commands in the Edge device and display the contents of the table.

    1. Launch InfluxDB.

      ~/modbus$ influx
      

      The following output is displayed if InfluxDB is launched successfully.

      Connected to http://localhost:8086 version 1.8.10
      InfluxDB shell version: 1.8.10
      
    2. Run the following command to select the sampleDB database.

      > USE sampleDB
      

      If the database selection is successful, the following output is displayed.

      Using database sampleDB
      
    3. Run the following command to view the stored data.

      > SELECT * FROM "sample_measurement" LIMIT 3
      

      Storage is successful if data is stored as shown below.

      name: sample_measurement
      time coil_1 holding_register_2 
      olding_register_3
      ---- ------ ------------------ ------------------
      1641876882000000000 false 23 13
      1641876887000000000 false 16 14
      1641876892000000000 true 15 13
      

Visualization and monitoring

Use Grafana to visualize the data that is stored in InfluxDB.

  1. Run the following command in Edge device and start Grafana.

    sudo systemctl start grafana-server
    
  2. In the address bar of your Internet browser, specify the following URL:

    <IP address of the device>:3000
    GrafanaLogin

  3. In the Username and Password boxes, type admin, and click Log in.

    Both the username and password has been configured to admin. You can change them as necessary.

Database registration

Follow these steps to register the sampleDB database that was created in the step Configuring the database, in Grafana.

  1. On the left pane, click the gear icon, navigate to Configuration > Data Sources, and then click Add data source.

    AddDataSource_grafana

  2. Select InfluxDB.

    Image description

    The settings page appears.

  3. Enter the following details and click Save & test.

    Setting name Setting value
    URL http://localhost:8086/
    Database sampleDB

    URL
    SaveandTest

Visualizing time-series data

In this example, we show you how to create a dashboard and display the collected time-series data in graphs such that:

  • Values of holding register 2 are displayed in line graphs as raw data
  • Values of holding register 3 are displayed as the average of values every minute

Follow these steps to create the dashboard:

  1. On the left pane, click the + icon, select Create > Dashboard, and then click Add a new panel.
    NewDashboard

  2. Configure the graph settings for the holding register 2 as shown in the screen below, and click Apply.
    GraphSettings

    Note: You can click the Edit icon and enter following query to configure query settings.
    SELECT "holding_register_2" FROM "sample_measurement" WHERE $timeFilter

  3. Repeat steps 1-2 to create a bar graph for holding register 3 (holding_register_3) and configure the settings as shown in the image below.

    HoldingRegister3_graph

    Note: You can also configure the query settings with the following query command.
    SELECT mean("holding_register_3") FROM "sample_measurement" WHERE $timeFilter GROUP BY time(1m)

Visualizing Boolean data

Follow these steps to display the latest status of coil 1.

  1. Click Add panel to add a panel.

  2. Select Stat from the right-side menu and configure the settings as shown in the image below.
    False

    Note: You can also configure query settings with the following query command.
    SELECT "coil_1" FROM "sample_measurement" WHERE $timeFilter

    When all the operations are completed, the following screen is displayed.
    True

  3. Click Save dashboard to save the dashboard.

Conclusion

This article described how to collect data through Modbus communication using Python, store the data in InfluxDB, and visualize data in Grafana on the Edge device.

This article is useful when you want to visualize data locally, without having to save the data in external systems.

References

  1. AI platform for industries | Yokogawa Electric Corporation
  2. Install on Debian/Ubuntu | Grafana Labs
  3. Install InfluxDB OSS | InfluxDB OSS 1.8 Documentation
  4. GitHub - riptideio/pymodbus: A full modbus protocol written in python

Discussion (0)