If your Sidekiq job is stuck, your first intention may be to try to stop this job. But Sidekiq doesn't have such mechanism, and it may make your worker stuck even futher, and to clean current processing job.
So the first thing that should be - to get logs about what is happening. To do that, send the TTIN signal to dump backtraces to the log, with command kill -TTIN [Sidekiq pid here. In docker container it most probably 1]
$ ps ax | grep sidekiq
PID TTY STAT TIME COMMAND
1 ? Ssl 94:02 sidekiq 5.2.9 app [1 of 20 busy]
$ kill -TTIN 1
It will dump the logs to stdout. Depending on your application settings, you may want to check logs on your logs collector service, e.g. in Kibana dashboard.
If the Sidekiq process is not responding, you can use GDB to dump backtraces for all threads with command gdb [ruby exec] [process number], like this:
$ which ruby
#=> /usr/local/bin/ruby
$ gdb /usr/local/bin/ruby 1
(gdb) info threads
Id Target Id Frame
37 Thread 0x7f8b289d8700 (LWP 7994) "ruby-timer-thr" 0x00007f8b27a20d13 in *__GI___poll (fds=<optimized out>, fds@entry=0x7f8b289d7ec0, nfds=<optimized out>,
nfds@entry=1, timeout=timeout@entry=100) at ../sysdeps/unix/sysv/linux/poll.c:87
36 Thread 0x7f8b23eb0700 (LWP 7995) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
35 Thread 0x7f8b23c2e700 (LWP 7996) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
34 Thread 0x7f8b239ac700 (LWP 7997) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
33 Thread 0x7f8b237aa700 (LWP 7998) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
32 Thread 0x7f8b28844700 (LWP 8002) "SignalSender" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
31 Thread 0x7f8b1e1bf700 (LWP 8003) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
30 Thread 0x7f8b1e0be700 (LWP 8006) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
29 Thread 0x7f8b1dd81700 (LWP 8009) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
28 Thread 0x7f8b1dc80700 (LWP 8010) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
27 Thread 0x7f8b1db7f700 (LWP 8011) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
...
(gdb) set logging file gdb_output.txt
(gdb) set logging on
(gdb) set height 10000
(gdb) t a a bt
(gdb) quit
It will save logs to gdb_output.txt.
You also can puts ruby logs to stdout, like so
(gdb) call (void)rb_backtrace()
After that, you can try to stop job in Sidekiq Admin with 'Stop' button, or with signals(TSTP to ask it to not pick up any new jobs, and TERM next).
Links
1 Sidekiq FAQ about frozen processes
2 Another guide about frozen processes
3 How to use GDB with ruby processes
4 How to stop Sidekiq jobs(as your application mechanic)
Top comments (0)