loading...
Cover image for Enabling Memory Profiling on KEDA v2

Enabling Memory Profiling on KEDA v2

tsuyoshiushio profile image Tsuyoshi Ushio ・Updated on ・3 min read

I have a minor memory leak issue to solve. So that I'm wondering how can we enable profiler on KEDA. KEDA uses Operator SDK.

Go lang has a good profiler.

It is very simple to use it. Just add this line for memory profiling.

Configuration and Issue

According to the documentation, the configuration will be like this.

go get github.com/pkg/profile
Enter fullscreen mode Exit fullscreen mode

then add following line to the main.go

import (
    //...
    "github.com/pkg/profile"
)
   :
defer profile.Start(profile.MemProfile).Stop()
Enter fullscreen mode Exit fullscreen mode

Run the KEDA on my local machine. However, it doesn't work. The root cause is, we need to stop the operator sdk with ctr+c. In this case, defer method doesn't execute Stop() method.

ushio@DESKTOP-KIUTRHV:~/Code/keda$ make run ARGS="--zap-log-level=debug"
/home/ushio/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go run \
-ldflags "-X=github.com/kedacore/keda/version.GitCommit=d0788b943ecf94f7f7c5b1a09f777cc3af6b746e -X=github.com/kedacore/keda/version.Version=v2" \
./main.go --zap-log-level=debug
2020/10/20 09:19:49 profile: memory profiling enabled (rate 4096), /tmp/profile174831365/mem.pprof
                   :
^C2020/10/20 09:20:02 profile: caught interrupt, stopping profiles
Exit 0 for profiling
make: *** [Makefile:96: run] Interrupt

$ ls -l /tmp/profile287657069/mem.pprof
-rw-r--r-- 1 ushio ushio 0 Oct 20 09:18 /tmp/profile287657069/mem.pprof
Enter fullscreen mode Exit fullscreen mode

Solution

If you want to handle the SIG interrupt, you can use signal.Notify method for handling the signal.

func SetupCloseHandler() {
    c := make(chan os.Signal)
    signal.Notify(c, os.Interrupt, syscall.SIGTERM)
    go func() {
        <-c
        fmt.Println("\r- Ctrl+C pressed in Terminal")
        DeleteFiles()
        os.Exit(0)
    }()
}
Enter fullscreen mode Exit fullscreen mode

I read the main.go they are also having the signal handling part. Let's change it to call profile.Stop() method.

func main() {
    // Memory profiling
    f := profile.Start(profile.MemProfile, profile.NoShutdownHook)
    :
    //  if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
    //      setupLog.Error(err, "problem running manager")
    //      os.Exit(1) // Profiling test
    //  }
    fmt.Println("-----SetupCloseHandler")
    setupSignalHandler := func() (stopCh <-chan struct{}) {
        stop := make(chan struct{})
        c := make(chan os.Signal, 2)
        signal.Notify(c, os.Interrupt, syscall.SIGTERM)
        go func() {
            <-c
            fmt.Println("Exit 0 for profiling")
            f.Stop()
            os.Exit(1) // second signal. Exit directly.
        }()

        return stop
    }

    if err := mgr.Start(setupSignalHandler()); err != nil {
        setupLog.Error(err, "problem running manager")
        os.Exit(1)
    }
}
Enter fullscreen mode Exit fullscreen mode

Run it. Works!

$ ls -l /tmp/profile388737932/mem.pprof
-rw-r--r-- 1 ushio ushio 47888 Oct 20 09:31 /tmp/profile388737932/mem.pprof
Enter fullscreen mode Exit fullscreen mode

Viewer

The next step will be the view the analysis report.
It looks requires graphviz. Install it.

$ go tool pprof --pdf /tmp/profile356144532/mem.pprof > file.pdf
failed to execute dot. Is Graphviz installed? Error: exec: "dot": executable file not found in $PATH
$ sudo apt install graphviz
Enter fullscreen mode Exit fullscreen mode

Works!

$ go tool pprof --pdf /tmp/profile356144532/mem.pprof > file.pdf
Enter fullscreen mode Exit fullscreen mode

Alt Text

I don't know if there is better way to enable profiling using Operator SDK. Once I know it, I'll update this blog.

Reference

Discussion

pic
Editor guide