How TLS / mTLS handshakes Work?

TLS/SSL (transport layer security)

TLS is a transport layer security protocol (the successor of the Secure Sockets Layer (SSL) protocol) and it secure the network communication in two ways:

• Encrypting the traffic against eavesdroppers on the internet
• Validating the authenticity of the counterparty

In this protocol, client applications prioritize verifying the identity of the server application. For instance, when entering an address like google.com in the web browser's address bar, it becomes crucial to ensure a secure connection with the legitimate google.com website, avoiding any potential connection to a different or malicious site. Here is a schematic flow of how a typical TLS connection is established:

In TLS (Transport Layer Security), the trust is established through a trusted certificate authority. This authority possesses a public-private key pair. The client applications have access to the public key of this certificate authority.
To secure the server's identity, the server generates its own public-private key pair. It then creates a Certificate Signing Request (CSR) that includes relevant information such as the website's domain name and the server's public key. This CSR is sent to the certificate authority for verification.
Upon receiving the CSR, the certificate authority validates the information provided, signs it using its private key, and issues a certificate for the server. This certificate contains the original information submitted by the server, along with a digital signature. This signature can be verified using the public key of the certificate authority.
These steps are essential prior to initiating a TLS handshake between the client application and the server, ensuring a trusted and secure connection. The next steps are as follows:

• The client say hello to the server
• Server replies and send it's certificate
• The client verifies the signature of the certificate using the CA public key.
• If verified, the client generate an ESA key, encrypt it with the public key of the server and send it back to the server. These key can only be read by the server, because it is encrypted with the public key of the server and can only be decrypted using the private key of the server.
• Subsequent messages communicated between client and the server will be encrypted with that ESA key. because only the client and the server have this key, only these two can decrypt the messages.

mTLS (mutual transport layer security)

Consider a scenario where both communicating parties desire to authenticate each other's identities. This is where the mTLS (Mutual Transport Layer Security) protocol comes into play, offering a solution. mTLS is a transport layer security protocol (like TLS) and it secure the network communication in two ways:

• Encrypting the traffic against eavesdroppers on the internet
• Validating the authenticity of the counterparty

In this protocol, the foundation of trust lies once again in a trusted certificate authority. For successful implementation, both communicating parties are required to register their identities with this trusted certificate authority. The handshake process resembles that of TLS, with the distinction that both sides must exchange their certificates with each other.