If you are contributing the KEDA, you might be wondering how we can debug it. I'd like to share the tips for debugging KEDA.
We have several ways to do it.
Remote Debugging
Remote debugging is not very recommended since it is relatively slow, however, sometimes, it requires. KEDA has two servers. one is keda-operator that has reconcile loop. It ask the infromation to the keda-metrics-apiserver that is only reside in the k8s cluster. In case you want to debug keda-metrics-apiserver we need the remote debugging.
Create a Dockerfile
Create a Dockerfile that enalbes us to debug. It is just adding delv and build with gcflags "-N -l". then start the execution file with wrapping with dlv command with exposing port.
ENTRYPOINT ["/dlv", "--listen=:4040", "--headless=true", "--api-version=1", "--continue", "--accept-multiclient", "exec", "/workspace/keda-adapter", "--", "--secure-port=6443", "--logtostderr=true", "--v=0"]
The parameters means following. -- means passing arguments to the executable (in this case, keda-adapter)
$ dlv --help
Delve is a source level debugger for Go programs.
Delve enables you to interact with your program by controlling the execution of the process,
evaluating variables, and providing information of thread / goroutine state, CPU register state and more.
The goal of this tool is to provide a simple yet powerful interface for debugging Go programs.
Pass flags to the program you are debugging using `--`, for example:
`dlv exec ./hello -- server --config conf/config.toml`
Usage:
dlv [command]
Available Commands:
attach Attach to running process and begin debugging.
connect Connect to a headless debug server.
core Examine a core dump.
dap [EXPERIMENTAL] Starts a TCP server communicating via Debug Adaptor Protocol (DAP).
debug Compile and begin debugging main package in current directory, or the package specified.
exec Execute a precompiled binary, and begin a debug session.
help Help about any command
run Deprecated command. Use 'debug' instead.
test Compile test binary and begin debugging program.
trace Compile and begin tracing program.
version Prints version.
Flags:
--accept-multiclient Allows a headless server to accept multiple client connections.
--api-version int Selects API version when headless. New clients should use v2. Can be reset via RPCServer.SetApiVersion. See Documentation/api/json-rpc/README.md. (default 1)
--backend string Backend selection (see 'dlv help backend'). (default "default")
--build-flags string Build flags, to be passed to the compiler.
--check-go-version Checks that the version of Go in use is compatible with Delve. (default true)
--headless Run debug server only, in headless mode.
--init string Init file, executed by the terminal client.
-l, --listen string Debugging server listen address. (default "127.0.0.1:0")
--log Enable debugging server logging.
--log-dest string Writes logs to the specified file or file descriptor (see 'dlv help log').
--log-output string Comma separated list of components that should produce debug output (see 'dlv help log')
--only-same-user Only connections from the same user that started this instance of Delve are allowed to connect. (default true)
--wd string Working directory for running the program. (default ".")
Additional help topics:
dlv backend Help about the --backend flag.
dlv log Help about logging flags.
Use "dlv [command] --help" for more information about a command.
Docker.remote
FROM golang:1.15 as builder
WORKDIR /workspace
RUN go get -u github.com/go-delve/delve/cmd/dlv
# Copy the Go Modules manifests
COPY go.mod go.mod
COPY go.sum go.sum
# cache deps before building and copying source so that we don't need to re-download as much
# and so that source changes don't invalidate our downloaded layer
RUN go mod download
COPY Makefile Makefile
# Copy the go source
COPY hack/ hack/
COPY version/ version/
COPY main.go main.go
COPY adapter/ adapter/
COPY api/ api/
COPY controllers/ controllers/
COPY pkg/ pkg/
RUN mkdir -p /apiserver.local.config/certificates && chmod -R 777 /apiserver.local.config
# Build
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -gcflags "-N -l" -o keda-adapter adapter/main.go
FROM ubuntu:18.04
WORKDIR /
COPY --from=builder /apiserver.local.config /apiserver.local.config
COPY --from=builder /workspace /workspace
COPY --from=builder /go/bin/dlv /dlv
ENTRYPOINT ["/dlv", "--listen=:4040", "--headless=true", "--api-version=1", "--continue", "--accept-multiclient", "exec", "/workspace/keda-adapter", "--", "--secure-port=6443", "--logtostderr=true", "--v=0"]
Then build and publish to your dockerhub account. In my case,
$ docker login
$ docker build -f Docker.remote -t tsuyoshiushio/adapter:1
$ docker push tsuyoshiushio/adapter:1
Update the deployment
We can see the two deployment is available on the k8s cluster.
ushio@DESKTOP-KIUTRHV:~/Code/keda$ kubectl get deployment -n keda
NAME READY UP-TO-DATE AVAILABLE AGE
keda-metrics-apiserver 1/1 1 1 6h2m
keda-operator 1/1 1 1 6h2m
let's edit the keda-metrics-apiserver.
$ kubectl edit deployment keda-metrics-apiserver -n keda
let's change the image part.
spec:
containers:
- args:
- /usr/local/bin/keda-adapter
- --secure-port=6443
- --logtostderr=true
- --v=0
env:
- name: WATCH_NAMESPACE
image: docker.io/tsuyoshiushio/adapter:3 // <- Change
imagePullPolicy: Always
then port forward it.
$ kubectl get pods -n keda
NAME READY STATUS RESTARTS AGE
keda-metrics-apiserver-6d5689599f-884x9 1/1 Running 0 68s
keda-operator-6b546dc696-cxdbq 1/1 Running 0 5h49m
$ kubectl port-forward keda-metrics-apiserver-6d5689599f-884x9 -n keda 4040:4040
Forwarding from 127.0.0.1:4040 -> 4040
Forwarding from [::1]:4040 -> 4040
For VS Code, you can configure launch.json settings to attach remote server.
{
"name": "Remote debug",
"type": "go",
"request": "attach",
"mode": "remote",
"remotePath": "/workspace",
"port": 4040,
"host": "127.0.0.1"
}
However, it is sometimes, hard to wait very slow response. I recommend to go with Local debugging and adding log to see the remote container.
Local Debugging
Set the launch.json with VSCode. Just start the main.go on the root of the KEDA repo. We can not use keda-metrics-apiserver since it reside in the cluster. However, we can do the local debugging for the keda-operator.
{
"name": "Connect to server",
"type": "go",
"request": "launch",
"mode": "auto",
"program": "${workspaceFolder}/main.go",
"env": {
"WATCH_NAMESPACE":""
}
},
Build and publish KEDA to your kubernets cluster
make undeploy will remove the keda resources. make deploy deploy the keda resources to the cluster when you in development.
$ kubectl get all -n keda
NAME READY STATUS RESTARTS AGE
pod/keda-metrics-apiserver-6dff8c4f7f-8qc9v 1/1 Running 0 30s
pod/keda-operator-6b546dc696-pf5ck 1/1 Running 0 29s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/keda-metrics-apiserver ClusterIP 10.0.16.80 <none> 443/TCP,80/TCP 31s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/keda-metrics-apiserver 1/1 1 1 30s
deployment.apps/keda-operator 1/1 1 1 29s
NAME DESIRED CURRENT READY AGE
replicaset.apps/keda-metrics-apiserver-6dff8c4f7f 1 1 1 31s
replicaset.apps/keda-operator-6b546dc696 1 1 1 30s
Top comments (0)