For the longest time, I wasn’t a fan of debuggers. Coming from a background in Spring Framework, Java, Python, and PHP (Symfony/Laravel), I always found logs and traces more reliable for debugging. I had even dabbled with GDB in the early days, but it didn’t stick. Instead, I relied on logging to figure out what my applications were doing.
However, my perspective changed when I started working with Go. While developing in Kubernetes and working on microservices, I faced a situation where the complexity of the service made logging and tests insufficient. This was a large service with multiple dependencies, serving both users and operators, making bug fixes particularly challenging.
Let me share a recent experience where using a debugger saved me a significant amount of time.
Debugging in Kubernetes: A Real-World Example
A few weeks ago, I was developing a feature that affected multiple parts of a microservice running in Kubernetes. This service was being called by several other components (like the app and admin panels), and I needed to trace what was happening inside the service, step by step. Tests couldn’t cover everything, especially when I needed to ensure that certain flags were set correctly to prevent sending multiple SMS notifications to users.
I could have used logs to trace the service, but every time I missed an entry, I would have had to add a new one, push the code, and wait for the CI pipeline to complete—wasting 3-5 minutes each time just to deploy to staging.
Instead, I decided to try using a debugger. Here’s how I set it up.
Setting Up Delve Debugger in a Kubernetes Pod
After a lot of trial and error, I found that adding the following lines to the Dockerfile allowed me to run Delve (a debugger for Go) inside the Kubernetes Pod:
ENTRYPOINT ["/bin/sh", "-c", "/go/bin/dlv --listen=127.0.0.1:8001 --headless=true --api-version=2 --only-same-user=false exec /path/to/exec"]
To install Delve, I added this line to the Dockerfile:
RUN go install github.com/go-delve/delve/cmd/dlv@v1.23.0
Next, I compiled the Go application with specific flags to disable optimizations and inlining:
go build -gcflags "all=-N -l"
Connecting to the Debugger
Once the service was deployed, I used port forwarding to connect to the debugger. I added the following port configuration to the Kubernetes manifest:
ports:
- name: dlv
containerPort: 8081
Then, I forwarded the port to my local machine:
oc port-forward $(oc get pods -l app=LABEL -o name | head -n 1) 8001:8001
Now I could connect to the debugger from my local machine using:
dlv connect :8001
Setting Breakpoints and Configuring Paths
With the debugger connected, I set breakpoints, such as:
(dlv) break main.main
Because the service was built in a different environment (GitLab’s pipeline), the paths didn’t match my local setup. To solve this, I used substitute-path to map the paths:
(dlv) config substitute-path /usr/local/go/ /Users/hossein/sdk/go1.23.1/
Running these commands every time was tedious, so I created a script (dlv.init) to automate the setup:
dlv connect :8001 --init dlv.init
Explanation of Key Commands
Delve Debugger Command:
dlv --listen=127.0.0.1:8001 --headless=true --api-version=2 --only-same-user=false exec /path/to/exec
-
--listen: Opens the debugger on127.0.0.1:8001for remote connections. -
--headless=true: Runs Delve without an interactive UI, suitable for remote debugging. -
--api-version=2: Uses API version 2 for better tool compatibility. -
--only-same-user=false: Allows users other than the process owner to connect. -
exec /path/to/exec: Starts or attaches to the Go executable for debugging.
This setup allows remote debugging of a Go service in a Kubernetes environment.
Go Build Command:
go build -gcflags "all=-N -l"
-
-N: Disables optimizations. -
-l: Prevents inlining of functions.
These flags ensure the code stays closer to the source, making it easier to debug by allowing you to step through every function.
Example Usage
To demonstrate how to debug a remote service, I use a local Docker container for simplicity. However, in practice, this service would be deployed on Kubernetes.
To build the Docker image, run the following command:
make build
This command will build a Docker image. Once built, you can start the service by running:
make start
Output:
$ make start
docker run -it -p 8001:8001 -p 8080:8080 --rm my-app
API server listening at: [::]:8001
2024-10-12T13:48:31Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
The prompt shows that the debugger is running. To attach the debugger to the remote service, run:
make debug
Output:
$ make debug
bash -c "dlv connect :8001 --init <(sed 's|PWD|'`pwd`'|g; s|HOME|'$HOME'|g' dlv.init)"
Type 'help' for list of commands.
Breakpoint 1 set at 0x2188cc for main.main.func1() ./main.go:9
(dlv) c
Once attached, send the c command to continue execution. Then, send a request to the service using the following:
make send-request
The debugger will show output like this:
> [Breakpoint 1] main.main.func1() ./main.go:9 (hits goroutine(17):1 total:1) (PC: 0x2188cc)
Warning: debugging optimized function
Warning: listing may not match stale executable
4: "fmt"
5: "net/http"
6: )
7:
8: func main() {
=> 9: http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
10: fmt.Fprintf(w, "Hello, World!")
11: })
12:
13: http.ListenAndServe(":8080", nil)
14: }
At this point, you can manually debug the service. For example, to inspect the HTTP method, use:
(dlv) p r.Method
"GET"
Once troubleshooting is complete, use the c command to continue and the client will receive the output.
Conclusion
Using a debugger in this case allowed me to step through the code and understand the service's behavior without constantly pushing new logging code. It was a game-changer, especially when working with complex microservices in Kubernetes. Debugging Go in production environments can be daunting, but with the right setup, it can save you a lot of time and frustration.
You can check out the full example of this setup on my GitHub.
Top comments (0)