Sherlock Holmes: The Case of the Missing User IPs

Welcome to our series of infrastructure detective stories, where we unravel the mysteries lurking within our systems In this episode, we tackle a perplexing problem: our Nginx server inside Kubernetes was logging an IP address that didn't match the actual user IP. Join me as we unravel the mystery and uncover the truth behind the missing IP.

The Mysterious IP

It all started when we noticed something unusual in our logs. The IP addresses recorded by Nginx were not matching the actual IPs of our users. Every log entry showed the same IP, regardless of the user. Clearly, something was amiss. The first clue led us to believe that a proxy was involved, obscuring the true origin of the requests.

The Proxy Puzzle

Determined to get to the bottom of this, I decided to log the X-Forwarded-For header, hoping it would reveal the original IP. However, to my surprise, the header was empty. This deepened the mystery. What could be interfering with our ability to see the user IPs?

The Gateway Revelation

Next, I examined the IP that was consistently logged. It turned out to be an address ending with ".1", which matched the default gateway of our Nginx service. This pointed the investigation towards Kubernetes. There was something about the Kubernetes configuration that was hidding the user ip.

Kubernetes Configuration Conundrum

Digging into the Kubernetes documentation, I discovered that our nginx service's externalTrafficPolicy was set to Cluster. By default, Kubernetes balances the load among different Nginx pods, proxying requests and masking the original IP. This explained why the user IPs were not appearing in our logs.

Here’s a simple diagram to illustrate the Cluster traffic policy:

[Client IP] ---> [Kubernetes Load Balancer] ---> [Nginx Pod 1]
                                        \
                                         \--> [Nginx Pod 2]

The Final Piece

To solve the problem, I changed the externalTrafficPolicy to Local. This configuration ensures that all requests are sent directly to the pods without being proxied by Kubernetes. While this meant giving up on load balancing, it allowed us to log the true user IPs.

With the mystery solved, our logs now accurately reflect the IP addresses of our users.

Stay tuned for our next adventure, where we continue to unravel the mysteries of the infrastructure world, one case at a time. Until then, keep your magnifying glasses handy and your curiosity alive.

Finally, if the article was helpful, please clap 👏and follow, thank you!