Implementation of an encrypted MQTT connection with MicroPython on a ESP32

Table of Contents

1. Introduction: ESP32 MicroPython MQTT TLS
2. Installing MicroPython
3. Using MicroPython
4. Wifi and MQTT Connection
5. MQTT and TLS

Introduction: ESP32 MicroPython MQTT TLS

by: Bass Paranoya

Overview

In this tutorial you will learn how to program the ESP32 using MicroPython. At first we will connect the device to the Internet via WIFI. Furthermore we will use this knowledge to implement a MQTT connection to send internal sensor data. In the last steps we will discuss data security and use it to authenticate and encrypt our MQTT connection over TLS.

Prerequisites and required Equipment

• Computer
• Visual Studio Code
• ESP32 DevKitC
• USB-Cable which allows to transfer data

The tutorial was made on a Windows system. Nevertheless, I will also present the commands for Ubuntu. If you use a Mac or another system you'll find the necessary commands most of the time in the presented external links (APIs) at the end of each lesson under "Hints and pitfalls", "Useful Resources for Own Searches" and "Sources"). I recommend to use Visual Studio Code.

Lesson 1: Installing MicroPython

Objectives

Our first lesson will be flashing the ESP32 with MicroPython.

Prerequisites and required Equipment

• ESP32 DevKitC
• USB-Cable which allows to transfer data
• Computer

Solution Steps

1. Download the latest MicroPython firmware:
   
   Click here for downloading the latest .bin file (e.g.: v1.17 (2021-09-02) .bin)

2. Connect the ESP32 with the USB cable to your computer.

3. Install the esptool extension using pip in a terminal or the powershell in VS Code:

Click here if you have not installed pip.

pip install esptool

• alternatively you can use this GitHub Link.

You need to look up the port used for the ESP32 in the Device Manager. Adjust the COMx for your command in the next step.

If the ESP32 does not connect to any port, the chance is high that your USB cable is not suitable for exchanging data (it is a power supply only). Use a suitable one instead.

1. Erase the flash under Windows with:

python -m esptool --port COMx erase_flash 

• Or on Linux

esptool.py --port /dev/ttyUSBx erase_flash

• If the command output says something similar to "no permission", then it is likely the ESP32 is connected somewhere else. Disconnect it.

1. Next we will deploy the new firmware with this example command on Windows which worked best on my system (use your port and path):

python -m esptool --chip esp32 --port COMx write_flash -z 0x1000 C:\path_to_firmware\esp32-20210902-v1.17.bin 

• If you use Linux:

esptool.py --chip esp32 --port /dev/ttyUSBx write_flash -z 0x1000 esp32-20180511-v1.9.4.bin

1. If the above commands run without error then MicroPython should be installed on your board.

Hints and pitfalls

• You may need to reduce the baudrate if you get errors while flashing (e.g. down to 115200 by adding --baud 115200 into the command) or (-b 9600 is a very slow value that you can use to verify it is not a baud rate problem)
• For some boards with a particular FlashROM configuration you may need to change the flash mode (e.g. by adding -fm dio into the command)
• Check if you have permissions to access the serial port, and if other software (such as modem-manager on Linux) is trying to interact with it. A common pitfall is leaving a serial terminal accessing this port open in another window and forgetting about it.
• for more tips check out the following links:

Useful Resources for Own Searches

Click here for the Getting Started with MicroPython on the ESP32 from the MicroPython API

Click here for the Installation and Dependencies tutorial from ESPRESSIF

Check out the Troubleshooting from ESPRESSIF if you still get Errors

What's next

Using MicroPython

Source(s)

• Micropython API for ESP32 Boards: https://docs.micropython.org/en/latest/esp32/quickref.html
• ESPRESSIF documentation: https://docs.espressif.com/projects/esptool/en/latest/esp32/index.html#

Lesson 2: Using MicroPython

Objectives

Next, we will get a REPL promt in which we can use MicroPython on the ESP32.

Prerequisites and required Equipment

• ESP32 DevKitC
• Computer
• USB-Cable which allows data transfer
• Visual Studio Code

Solution Steps

1. In order to use Pymakr extension on VS Code, you need node.js installed on your computer. Use this link to get to the download page.

2. Open Visual Studio Code and install the Pymakr Extension (ID: pycom.pymakr)

3. If you have any trouble, here is a detailed manual (at the top of the page you can also follow the manual for installing VS Code)

4. Edit the pymakr.json file which opens automatically after installing Pymakr, in case it did not: Hit the "ALL commands" button at the very bottom of the VS Code window and click on "Global Setting".

• Pymakr has a list of USB devices it will connect to. You need to make sure that your device is on the list in the pymakr.json configuration file. You need to add the USB Manufacturer that the device uses to connect to the PC. In our case, the ESP32 uses the "Silicon Labs" USB drivers.

• To find out which drivers your board uses, open your computer’s Device Manager (with the board connected to your computer) and search for connected USB devices. Under "General" you find the "Manufacturer". Now, edit the file to add the Silicon Labs manufacturer (or other) to the autoconnect_comport_manufacturers
  
  section. Then, save the file. The section should look like this:
  

"autoconnect_comport_manufacturers": [
  "Pycom",
  "Pycom Ltd.",
  "FTDI",
  "Microsoft",
  "Microchip Technology, Inc.",
  "Silicon Labs"
]

1. Connect your flashed ESP32 to your computer and hit the "Pymakr Console" button at the very bottom left of the VS Code window. Navigate to the Pymakr Console (similar to powershell in VS Code). You will get the MicroPython REPL indicated by >>>.

1. You can now test the REPL promt by entering:

print('Hallo Studierende!')

1+2

• Your output should look like this:

>>> print('Hallo Studierende!')
Hallo Studierende!
>>> 1+2
3
>>>

• If it does, congratulations, you are using MicroPython on your ESP32 🎉🥳 <!--- einfach reinkopieren z.B. von https://emojipedia.org/party-popper/ --->

For a Warm-Up try:

>>> f = open('data.txt', 'w')
>>> f.write('some data')
9
>>> f.close()
>>> 
>>> f = open('data.txt')
>>> f.read()
'some data'
>>> f.close()
>>> 
>>> import os
>>> os.listdir()
['boot.py', 'main.py', 'data.txt']
>>> 
>>> os.mkdir('dir')
>>> 
>>> os.remove('data.txt')
>>> os.remoce("dir")

• There are two files that are treated specially by the ESP32 (and ESP8266) when it starts up: boot.py and main.py. The boot.py script is executed first (if it exists) and then once it is completed the main.py script is executed. You can create these files yourself and populate them with the code that you want to run when the device starts up.

REPL promt tips:

• Pressing ctrl-E will enter a special paste mode. This allows you to copy and paste a chunk of text into the REPL. If you press ctrl-E you will see the paste-mode prompt:

paste mode; Ctrl-C to cancel, Ctrl-D to finish
===

• You can then paste (or type) your text in. Note that none of the special keys or commands work in paste mode (e.g. Tab or backspace), they are just accepted as-is. Press ctrl-D to finish entering the text and execute it.

• There are four other control commands:

  • Ctrl-A on a blank line will enter raw REPL mode. This is like a permanent paste mode, except that characters are not echoed back.
  • Ctrl-B on a blank like goes to normal REPL mode.
  • Ctrl-C cancels any input, or interrupts the currently running code.
  • Ctrl-D on a blank line will do a soft reset.

Hints and pitfalls

• Regarding pymakr.json: put your COMx in the address setting and set auto_correct to False.
• Regarding pymakr.json: For automatic connect, let the address setting empty, and set the auto_connect to True. Play around with these settings if you encounter problems.
• If you can't get the Pymakr Console after opening VS Code make sure your computer is not running on a battery saving mode or similar. Set it to high performance mode and open VS Code again.
• If you cannot get the REPL promt indicated by >>>, perform a keyboard interrupt or click the "Upload" or "Run" button at the very bottom of the VS Code window. Play around, that was sometimes necessary on my system.
• If you cannot connect your ESP32, you can try it with TeraTerm or picocom on Linux. Read here for a detailed tutorial. Don not forget to set the baudrate to 115200.

Useful Resources for Own Searches

Click here for a detailed tutorial on how to install VS Code and the Pymakr extension

Click here for a MicroPython tutorial for the ESP8266 (most of them also work on your ESP32)

What's next

Setting up you Wifi connection and connect to a public test MQTT broker.

Source(s)

• Micropython API: https://docs.micropython.org/en/latest/index.html
• Random nerd tutorials: https://randomnerdtutorials.com/micropython-esp32-esp8266-vs-code-pymakr/#pymakr

Lesson 3: Wifi and MQTT Connection

Objectives

In this lesson you will set up a connection to your Wifi. Followed by connecting your ESP32 as a client to a public testing broker. Moreover we will subscribe to a topic and publish some data to get familiar with MQTT.

Prerequisites and required Equipment

• ESP32 DevKitC
• Computer
• USB-Cable which allows data transfer
• Visual Studio Code

Solution Steps

1. First, we will implement an automatic Wifi connection. This is done by writing the code into the boot.py file. Create a new file in the explorer (from VS Code on the left side of the window and name it boot.py). The code in this file will be executed automatically after every start of the module. (For testing purposes you can also first enter the code one by one into the REPL promt and see if your connection works.) To do so write the following lines (in the boot.py):

import network

ssid = 'Your SSID'
password = 'Your password' 

station = network.WLAN(network.STA_IF) # creates station interface

station.active(True) # activates the interface
station.connect(ssid, password) # connects to wifi

while station.isconnected() == False:
  pass # waits till module is connected

print('Connection successful')
print(station.ifconfig()) 
# prints the interface's IP/netmask/gw/DNS addresses

• Save the boot.py file and click the "Upload" button at the very bottom of the VS Code window. The whole project will be uploaded to your ESP32 and executed afterwards. If you just want to upload the boot.py file you can do that by clicking -> "All commands" -> "Upload current file only" (at the bottom of VS Code).

• When executed successfully, your terminal should print something like: Connection successful
('194.xxx.xxx.xxx', '254.xxx.xxx.xxx', '194.xxx.xxx.xxx', '194.xxx.xxx.xxx')

  If that is the case, congratulations 🎉 you connected the ESP32 with your Wifi.

  Lets move on with MQTT:

1. Create a new file named umqttsimple.py and copy the umqttsimple library code into it. You can access the umqttsimple library code via the following link:

   https://raw.githubusercontent.com/RuiSantosdotme/ESP-MicroPython/master/code/MQTT/umqttsimple.py

   Don't forget to save the file.

2. Moving on you will have to extend the boot.py by adding the following lines:
   

import time
from umqttsimple import MQTTClient
import ubinascii
import machine
import micropython
import esp
import esp32          # import all needed libraries
esp.osdebug(None)     # debug set to none
import gc             # garbage collector
gc.collect()

mqtt_server = 'YOUR MQTT BROKER'
#mqtt_server = "broker.emqx.io"     # works best for me
#mqtt_server = "broker.hivemq.com"
#mqtt_server = '3.65.154.195:1883'  #broker.hivemq.com
#mqtt_server = '5.196.95.208'       #test.mosquitto.org

client_id = ubinascii.hexlify(machine.unique_id()) # gets ID
topic_sub = b'OnBoard_Temp' # toppic name

last_message = 0 
message_interval = 5
counter = 0             # variables needed for improved timing

• Don't forget to save.

  • The last_message variable will hold the last time a message was sent. The message_interval is the time between each message sent. Here we are setting it to 5 seconds (this means a new message will be sent every 5 seconds). The counter variable is simply a counter to be added to the message.

1. We will create the main.py now. To do so, copy the following code into it:

# Callback func handles what happens when message is received on a certain topic
def sub_cb(topic, msg):
  print((topic, msg))

# responsible for connecting to broker as well as to subscribe to a topic
def connect_and_subscribe():
  global client_id, mqtt_server, topic_sub
  client = MQTTClient(client_id, mqtt_server, port = 1883, user=b"emqx", password=b"0815")

  client.set_callback(sub_cb)
  client.connect()
  client.subscribe(topic_sub)
  print('Connected to %s MQTT broker, subscribed to %s topic' % (mqtt_server, topic_sub))
  return client

def restart_and_reconnect():
  print('Failed to connect to MQTT broker. Reconnecting...')
  time.sleep(10)
  machine.reset()

try:
  client = connect_and_subscribe()
except OSError as e:
  restart_and_reconnect()

while True:
  try:
    client.check_msg()
    if (time.time() - last_message) > message_interval:
      tf = esp32.raw_temperature()
      tc = round((tf-32.0)/1.8, 2)
      msg = b'Message #%d' % (counter) + ": Temperature: %s degree Celsius" % (tc)
      client.publish(topic_sub, msg)
      last_message = time.time()
      counter += 1
  except OSError as e:
    restart_and_reconnect()

• Don't forget to save.

• We will receive and publish messages in the while loop. We use try and except statements to prevent the ESP from crashing in case something goes wrong.

• Inside the try block we start by applying the check_msg() method on the client.

• The check_msg() method checks whether a pending message from the server is available. It waits for a single incoming MQTT message and process it. The subscribed messages are delivered to the callback function we have defined earlier (the sub_cb() function). If there is not a pending message, it returns with None.

• Then we add an if statement to check whether 5 seconds (message_interval) have passed since the last message was sent.

• We prepare the data we want to send. In our case I have chosen the internal temperature (there is also an internal hall sensor). We format the data as we need it and put it in the msg variable.

• To publish a message on a certain topic, you just need to apply the publish() method on the client and pass as arguments the topic and the message.

• After sending the message, we update the last time a message was received by setting the last_message variable to the current time.
  Finally, we increase the counter variable in every loop.
  If something unexpected happens, we call the restart_and_reconnect() function.

1. Finally, we can upload the files to the ESP32 ("Upload" button at the bottom). Your device will execute the code automatically when the upload succeeds.

1. If your output looks like in the picture above, you are done with this lesson 🥳💃🏽🕺🏽

Question for you: What temperature is measured?

Hints and pitfalls

• You may need to open the port 1883 in your firewall.

• As you may have noticed, I am giving you a few options to play with regarding the mqtt_server variable. On my system the "broker.emqx.io" works best. Additionally I encountered that the IP address written in letters works "better" than written in numbers.

• Connecting to a MQTT broker worked on my system after providing a user and password within the MQTTClient(). What you write in those variables is not crucial for now.

• When your device will not connect, try to connect to test.mosquitto.org via the terminal.

• When your device will not connect but connecting to test.mosquitto.org in the terminal(s) worked, you can try to make it work with Arduino first...

Useful Resources for Own Searches

Click here to get on the Good Foot with MicroPython on the ESP32

Click here for another approach for streaming data from ESP32 using MicroPython and MQTT (umqtt.robust)

Click here for testing mosquitto in the terminal(s)

Click here for a Arduino tutorial

What's next

Encrypted connection to a MQTT broker.

Further Inputs

Play around with the hall sensor data or maybe you have another external sensor you can connect to your ESP32.

For that check out the possibilities the ESP32 has to offer:

Source(s)

• MicroPython Hackathon: https://micropython-iot-hackathon.readthedocs.io/en/latest/mqtt.html

• Random nerd tutorials 2: https://randomnerdtutorials.com/micropython-mqtt-esp32-esp8266/

• Quick reference for the ESP32 - Micropython API: https://docs.micropython.org/en/latest/esp32/quickref.html#networking

Lesson 4: MQTT and TLS

Objectives

In this final lesson you will establish a secured connection with a public test broker. MQTT provides security, but it is not enabled by default. MQTT has the option for Transport Layer Security (TLS) encryption, just as used with HTTPS. It is recommended using that for any system you put into production.
MQTT also provides username/password authentication. Note that the password is transmitted in clear text. Thus, be sure to use TLS encryption if you are using authentication. In our case we already provided the authentication (unencrypted). Nevertheless, it was just to improve the system flow. Another popular way of authenticating is via certificates and can be used in addition to using user name and password authentication. Refer to this explanation in order to learn more about TLS/SSL.

Prerequisites and required Equipment

• ESP32 DevKitC
• Computer
• USB-Cable which allows data transfer
• Visual Studio Code

Solution Steps

1. We first need to install OpenSSL in order to create our certificates and keys. Click here for GitHub or here for the exe.

2. Create CA key pair: Navigate to the Windows start and search OpenSSL. Hit enter on "OpenSSL Command Promt". Make sure you run the following commands as administrator.
   

openssl genrsa -des3 -out ca.key 2048

genrsa: generates a RSA private key

des3: Using DES3 cipher for the key generation

out: specifies the output file name (.key)

2048: number of bits for the private key

• Your output should look like this:

C:\Users\schue>openssl genrsa -des3 -out ca.key 2048
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:

C:\Users\schue>

• Enter any password. But remember it, we will need it in a moment again.

• The pass phrase is used to protect the private key. The generated private file ca.key has both the private and public key.

1. Create CA certificate: Next we are creating a certificate for the CA, using the key pair created in the step before:

openssl req -new -x509 -days 1826 -key ca.key -out ca.crt

req: certificate request and certification utility

new: generate new certificate, it will prompt user for several input fields

x509: create a self signed certificate

days: specify the number of days the certificate is valid

key: key file with private key to be used for signing

out: specifies the file name for the certificate (.crt)

• You should get something like this:

C:\Users\schue>openssl req -new -x509 -days 3650 -key ca.key -out ca.crt
Enter pass phrase for ca.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:DE
State or Province Name (full name) [Some-State]:Bavaria
Locality Name (eg, city) []:Munich
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Uni
Organizational Unit Name (eg, section) []:Master
Common Name (e.g. server FQDN or YOUR name) []:schue
Email Address []:.

C:\Users\schue>

As Common Name use your user name like "schue" in my case.

1. Create broker key pair: Next, we are creating a private key for the server with:

openssl genrsa -out server.pem 2048

genrsa: generate a RSA private key

out: specifies the output file name (.pem)

2048: number of bits for the private key

1. Create certificate request from CA: That key needs to be certified, so we create a certificate request for it, and the certificate needs to be signed by the CA:

openssl req -new -out server.csr -key server.pem

req: certificate request and certification utility

new: create new request file file

out: file name for the certificate signing request (.csr)

key: file name of the key to be certified

• Your output should look like this:

C:\Users\schue>openssl req -new -out server.csr -key server.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:DE
State or Province Name (full name) [Some-State]:Bavaria
Locality Name (eg, city) []:Munich
Organization Name (eg, company) [Internet Widgits Pty Ltd]:UniMuni
Organizational Unit Name (eg, section) []:EL
Common Name (e.g. server FQDN or YOUR name) []:schue
Email Address []:.

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:0815
An optional company name []:IT

C:\Users\schue>

1. Verify and sign the certificate request: The last step with OpenSSL is to sign the server request through the CA to get the broker certificate:

openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out cert.der -days 360

x509: certificate display and signing utility

req: a certificate request is expected as input

in: input file for the certificate

CA: specifies the file to be signed

CAkey: CA private key to sign the certificate with

Cacreateserial: the serial number file gets created if it does not exist

out: output file name

days: how long the certificate shall be valid

1. Convert your .pem file to .der file:

openssl rsa -inform pem -in server.pem -outform der -out key.der

1. Finally move the created files to the ESP32: To do so get mpfshell:

Click here if the command below does not work.

pip3 install mpfshell

• Make sure your device is not connected via a Pymakr console or anywhere else. Then enter (on Windows) (and insert YOUR port number):

mpfshell -c "open COMx"

• and on Linux:

mpfshell -c "open ttyUSBx"

• Now you can list the files on the device with:

mpfs> ls

• To upload e.g. the local file boot.py to the device use:

mpfs> put boot.py

• You need to navigate to the directory (cd) which contains the boot.py first.

Get more mpf commands here

1. Lastly, we can adjust our code to use the "secured" connection (we are still connectiong to a public test broker! But you get the main idea, in case you would like to implement a TLS connection to your own broker).

• Add the following lines to your boot.py:

with open('key.der') as f:
  key_data = f.read()
with open('cert.der') as f:
  cert_data = f.read()

• Change the connect_and_subscribe() in your main.py:

def connect_and_subscribe():
  global client_id, mqtt_server, topic_sub, key_data, cert_data
  client = MQTTClient(client_id, mqtt_server, port = 8883, user=b"emqx", password=b"0815"
                      , ssl=True, ssl_params={'key': key_data, 'cert': cert_data})
  client.set_callback(sub_cb)
  client.connect()
  client.subscribe(topic_sub)
  print('Connected to %s MQTT broker, subscribed to %s topic' % (mqtt_server, topic_sub))
  return client

My output:

Uploading project (main folder)...
Not safe booting, disabled in settings

Uploading to /...
Reading file status
No files to upload
Upload done, resetting board...
OKets Jun  8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:5656
load:0x40078000,len:12696
load:0x40080400,len:4292
entry 0x400806b0
Connection successful
('194.xxx.xxx.xxx', '254.xxx.xxx.xxx', '194.xxx.xxx.xxx', '194.xxx.xxx.xxx')
Connected to broker.emqx.io MQTT broker, subscribed to b'OnBoard_Temp' topic
(b'OnBoard_Temp', b'Message #0: Temperature: 40.0 degree Celsius')
(b'OnBoard_Temp', b'Message #1: Temperature: 40.0 degree Celsius')
(b'OnBoard_Temp', b'Message #2: Temperature: 40.0 degree Celsius')
(b'OnBoard_Temp', b'Message #3: Temperature: 40.0 degree Celsius')

• Yeay, we are publishing our data over TLS.

Hints and pitfalls

• You may need to open the port 8883 in your firewall.

• You can move the cert files to your VS Code workspace and hit the upload button. But it will not upload the certificates to your device!

• You can also use ampy to move files to the ESP32. This is the most common way online. Nevertheless, ampy did not work on my system.

• Since my colleagues encountered problems with mpfshell and ampy on their systems, I am giving you a last option: You can use the uPyCraft IDE to transfer files to your ESP32. Installation instructions for: Windows, MacOS X and Linux.

Useful Resources for Own Searches

Click here for another introduction to TLS

Click here for another approach for streaming data from ESP32 using MicroPython and MQTT (umqtt.robust)

ENOENT ERROR

Further Inputs

You can also encrypt your message before sending it. In our case it will not work. Can you find out why? It would work with a separate broker (with own code). Here are two useful links to begin with:
https://forum.micropython.org/viewtopic.php?t=6726
https://docs.micropython.org/en/latest/library/cryptolib.html

Steve: "Encrypting the MQTT payload rather than the link has the advantage that the data is encrypted end to end and not just between the broker and the client.

It also means that the intermediate brokers don’t need to support SSL and that you don’t need to obtain and install certificates."

Unknown: "It is always a good idea to use further security mechanisms such as payload encryption, payload signature verifications, and authorization mechanisms. In general, more security never hurts."

Do you agree?
What do you think about these quotations?

Congratulations! You are at the end of my tutorial. Maybe you would like to set up your own broker on a Rpy next?

Source(s)

• Enable Secure Communication with TLS and the Mosquitto Broker: https://mcuoneclipse.com/2017/04/14/enable-secure-communication-with-tls-and-the-mosquitto-broker/

• Mosquitto SSL Configuration: http://www.steves-internet-guide.com/mosquitto-tls/

• OpenSSL Command Line Utilities: https://wiki.openssl.org/index.php/Command_Line_Utilities