DEV Community

Cover image for Enabling KEDA debugging

Enabling KEDA debugging

tsuyoshiushio profile image Tsuyoshi Ushio ・5 min read

If you are contributing the KEDA, you might be wondering how we can debug it. I'd like to share the tips for debugging KEDA.

We have several ways to do it.

Remote Debugging

Remote debugging is not very recommended since it is relatively slow, however, sometimes, it requires. KEDA has two servers. one is keda-operator that has reconcile loop. It ask the infromation to the keda-metrics-apiserver that is only reside in the k8s cluster. In case you want to debug keda-metrics-apiserver we need the remote debugging.

Create a Dockerfile

Create a Dockerfile that enalbes us to debug. It is just adding delv and build with gcflags "-N -l". then start the execution file with wrapping with dlv command with exposing port.

ENTRYPOINT ["/dlv", "--listen=:4040", "--headless=true", "--api-version=1", "--continue", "--accept-multiclient", "exec", "/workspace/keda-adapter", "--", "--secure-port=6443", "--logtostderr=true", "--v=0"]
Enter fullscreen mode Exit fullscreen mode

The parameters means following. -- means passing arguments to the executable (in this case, keda-adapter)

$ dlv --help
Delve is a source level debugger for Go programs.

Delve enables you to interact with your program by controlling the execution of the process,
evaluating variables, and providing information of thread / goroutine state, CPU register state and more.

The goal of this tool is to provide a simple yet powerful interface for debugging Go programs.

Pass flags to the program you are debugging using `--`, for example:

`dlv exec ./hello -- server --config conf/config.toml`

  dlv [command]

Available Commands:
  attach      Attach to running process and begin debugging.
  connect     Connect to a headless debug server.
  core        Examine a core dump.
  dap         [EXPERIMENTAL] Starts a TCP server communicating via Debug Adaptor Protocol (DAP).
  debug       Compile and begin debugging main package in current directory, or the package specified.
  exec        Execute a precompiled binary, and begin a debug session.
  help        Help about any command
  run         Deprecated command. Use 'debug' instead.
  test        Compile test binary and begin debugging program.
  trace       Compile and begin tracing program.
  version     Prints version.

      --accept-multiclient   Allows a headless server to accept multiple client connections.
      --api-version int      Selects API version when headless. New clients should use v2. Can be reset via RPCServer.SetApiVersion. See Documentation/api/json-rpc/ (default 1)
      --backend string       Backend selection (see 'dlv help backend'). (default "default")
      --build-flags string   Build flags, to be passed to the compiler.
      --check-go-version     Checks that the version of Go in use is compatible with Delve. (default true)
      --headless             Run debug server only, in headless mode.
      --init string          Init file, executed by the terminal client.
  -l, --listen string        Debugging server listen address. (default "")
      --log                  Enable debugging server logging.
      --log-dest string      Writes logs to the specified file or file descriptor (see 'dlv help log').
      --log-output string    Comma separated list of components that should produce debug output (see 'dlv help log')
      --only-same-user       Only connections from the same user that started this instance of Delve are allowed to connect. (default true)
      --wd string            Working directory for running the program. (default ".")

Additional help topics:
  dlv backend Help about the --backend flag.
  dlv log     Help about logging flags.

Use "dlv [command] --help" for more information about a command.
Enter fullscreen mode Exit fullscreen mode


FROM golang:1.15 as builder

WORKDIR /workspace

RUN go get -u

# Copy the Go Modules manifests
COPY go.mod go.mod
COPY go.sum go.sum
# cache deps before building and copying source so that we don't need to re-download as much
# and so that source changes don't invalidate our downloaded layer
RUN go mod download

COPY Makefile Makefile

# Copy the go source
COPY hack/ hack/
COPY version/ version/
COPY main.go main.go
COPY adapter/ adapter/
COPY api/ api/
COPY controllers/ controllers/
COPY pkg/ pkg/

RUN mkdir -p /apiserver.local.config/certificates && chmod -R 777 /apiserver.local.config

# Build
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -gcflags "-N -l" -o keda-adapter adapter/main.go

FROM ubuntu:18.04

COPY --from=builder /apiserver.local.config  /apiserver.local.config
COPY --from=builder /workspace /workspace
COPY --from=builder /go/bin/dlv /dlv

ENTRYPOINT ["/dlv", "--listen=:4040", "--headless=true", "--api-version=1", "--continue", "--accept-multiclient", "exec", "/workspace/keda-adapter", "--", "--secure-port=6443", "--logtostderr=true", "--v=0"]
Enter fullscreen mode Exit fullscreen mode

Then build and publish to your dockerhub account. In my case,

$ docker login
$ docker build -f Docker.remote -t tsuyoshiushio/adapter:1
$ docker push tsuyoshiushio/adapter:1
Enter fullscreen mode Exit fullscreen mode

Update the deployment

We can see the two deployment is available on the k8s cluster.

ushio@DESKTOP-KIUTRHV:~/Code/keda$ kubectl get deployment -n keda
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
keda-metrics-apiserver   1/1     1            1           6h2m
keda-operator            1/1     1            1           6h2m
Enter fullscreen mode Exit fullscreen mode

let's edit the keda-metrics-apiserver.

$ kubectl edit deployment keda-metrics-apiserver -n keda
Enter fullscreen mode Exit fullscreen mode

let's change the image part.

      - args:
        - /usr/local/bin/keda-adapter
        - --secure-port=6443
        - --logtostderr=true
        - --v=0
        - name: WATCH_NAMESPACE
        image:    // <- Change
        imagePullPolicy: Always
Enter fullscreen mode Exit fullscreen mode

then port forward it.

$ kubectl get pods -n keda
NAME                                      READY   STATUS    RESTARTS   AGE
keda-metrics-apiserver-6d5689599f-884x9   1/1     Running   0          68s
keda-operator-6b546dc696-cxdbq            1/1     Running   0          5h49m
$ kubectl port-forward keda-metrics-apiserver-6d5689599f-884x9  -n keda 4040:4040
Forwarding from -> 4040
Forwarding from [::1]:4040 -> 4040
Enter fullscreen mode Exit fullscreen mode

For VS Code, you can configure launch.json settings to attach remote server.

            "name": "Remote debug",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "remotePath": "/workspace",
            "port": 4040,
            "host": ""
Enter fullscreen mode Exit fullscreen mode

However, it is sometimes, hard to wait very slow response. I recommend to go with Local debugging and adding log to see the remote container.

Local Debugging

Set the launch.json with VSCode. Just start the main.go on the root of the KEDA repo. We can not use keda-metrics-apiserver since it reside in the cluster. However, we can do the local debugging for the keda-operator.

            "name": "Connect to server",
            "type": "go",
            "request": "launch",
            "mode": "auto",
            "program": "${workspaceFolder}/main.go",
            "env": {
Enter fullscreen mode Exit fullscreen mode

Build and publish KEDA to your kubernets cluster

make undeploy will remove the keda resources. make deploy deploy the keda resources to the cluster when you in development.

$ kubectl get all -n keda
NAME                                          READY   STATUS    RESTARTS   AGE
pod/keda-metrics-apiserver-6dff8c4f7f-8qc9v   1/1     Running   0          30s
pod/keda-operator-6b546dc696-pf5ck            1/1     Running   0          29s

NAME                             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
service/keda-metrics-apiserver   ClusterIP   <none>        443/TCP,80/TCP   31s

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/keda-metrics-apiserver   1/1     1            1           30s
deployment.apps/keda-operator            1/1     1            1           29s

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/keda-metrics-apiserver-6dff8c4f7f   1         1         1       31s
replicaset.apps/keda-operator-6b546dc696            1         1         1       30s
Enter fullscreen mode Exit fullscreen mode

Discussion (0)

Editor guide