function assertAuthorized() {
    user = db.findUser(request.getUsername(), hash(request.getPassword()));
    assert user != null;
}

function prepare() {
    assertVariables();
    assertAuhorized();
}

function doJob() {
    prepare();
    task1();
}

Thread main
  at HttpRequest.getUserName
  at prepare
  at doJob

(notice the `nsleep` and `nanosleep` calls - it's the OS making the actual sleep call)

and a process currently running in userspace will not be executing anything in the kernel space, so its stack will look similar to

(notice the `wait` and `poll` calls - the OS is waiting for tasks to run in the kernelspace).

3. For a **userspace stack**, you have 2 basic options:
    - use the `gdb` debugger to attach to a running process (`gdb -p ${pid}` and run `bt` inside)
    - install tools that can automate that for you, like `pstack` or `eu-stack` (comes with the _elfutils_ package)

`pstack` is usually preinstalled and it's very straightforward to use it -- just give it a PID and see the stack appear in your console. However, I prefer `eu-stack`, because it has more options, like thread printing, verbosity and other nice features (see `eu-stack --help`).

or with all the fancy options on:

../sysdeps/unix/sysv/linux/clock_nanosleep.c:78:5

/build/glibc-SzIz7B/glibc-2.31/posix/nanosleep.c:27:13

../csu/libc-start.c:308:16

The nice thing about userspace STs is that you don't need `sudo` to get them if the process is running under your user.

#### Windows
> sigh...
> ...

Probably the easiest approach is through the magnificent [ProcessExplorer](https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer), described [here](https://superuser.com/questions/462989/how-can-i-view-the-call-stack-of-a-running-process-thread) and in lots of other places across the internet.

- download and install ProcessExplorer
- launch ProcessExplorer
- find the PID you want an ST of
- right-click → properties
- select the _Threads_ tab
- select the thread you'd like to inspect
- click on the `Stack` button

The stack frames will look cryptic, but if that's good enough for you -- there you go! If you want a clearer ST, you'll have to download and install [Debug Tools](https://docs.microsoft.com/en-us/windows-hardware/drivers/download-the-wdk) and load Debug Symbols into the ProcessExplorer as shown in tutorials:
- Options → Configure Symbols
- `C:\Windows\system32\dbghlp.dll` as the DbgHelp DLL path (or whatever it is in your system)
- `SRV*C:\Symbols*http://msdl.microsoft.com/download/symbols` as the Symbols Path (or whatever it is in your system)


### JVM
Once again, in the case of JVM, there are at least 2 levels of call traces:
- native
- application

To get a native process ST please look above, as it's already been discussed.

To get an Application StackTrace, JVM provides an option to capture a ThreadDump (TD). There are a few ways to capture TDs of a running application:
- `jstack -l <pid>` - requires a JDK utility `jstack` to be available on the same machine the JVM is running on. This is the preferred approach in almost all cases. It prints the TD to the `jstack`'s output
- `jcmd <pid> Thread.print` - just like the above: requires JDK utility to be present and prints TD to the `jcmd`'s output. since it's more verbose and may be more cumbersome to remember, it's not a widely used technique
- `kill -3 <pid>` or `kill -s QUIT <pid>` (both are identical) - does not require any additional software, as `kill` usually comes with the Linux OS. Sending a SIGQUIT signal to the JVM causes it to print a TD to the JVM's output. And that's the main inconvenience, compared to `jstack` and `jcmd` approaches -- you don't get the TD _here_ and _now_ - you have to find the JVM's output (could be either a log file or a console) and extract the TD from there. It can get very messy... But sometimes that's the only option you have.

Capturing a TD in a JVM stops the JVM for a split second and resumes it immediately after. In some cases, the JVM might appear stuck and `jstack` may recommend using the `-F` option. As always, handle _the force_ with great care, as it might lock all your application threads for good (ask me how I know it).

### Other runtimes
Some runtimes have their own alternatives to JVM's `jstack`, others - either don't or support plugins for GNU's `gdb` to acquire TDs through a debugger. Refer to your runtime's documentation to see if they have any means to capture a call trace. Asking Google for a second opinion doesn't hurt either.

## Analysis
CallTrace analysis can provide us with a lot of information about what's going on with the application:

- how many threads are doing X at the moment
- how many threads are there
- which actions are "hanging" and at which function calls
- which threads are slacking
- which threads are stuck in infinite loops and which functions have caused them
- which threads are likely to be causing high CPU usage; what are they doing to cause that
- which threads are making outgoing long-lasting IO calls; where do these calls originate from in code
- which threads are blocking other threads (using synchronization primitives) and at which function calls (both blocked and blocking)
- which/how many threads are running in X part of your code (e.g. `PaymentService2.transfer()`), should you care to know if a particular method/function is being called
- which threads are likely to blow the Stack up (cause stack overflows); and through which function calls
- how large are your thread pools and whether they are over/undersized
- other useful info, depending on what's the problem


### Individual STs
Analysing individual plain-text StackTraces might seem challenging at first, but a few stacks later the pattern becomes easier to understand. However, in cases of ThreadDumps, especially with lots of running threads (hundreds-thousands), the manual analysis still is a difficult task to take on.

#### Find the right threads
The approach of manual ST analysis depends on what issue are we trying to tackle. Usually, we're focusing on busy threads - the ones with long call traces. However, "long" is a relative term. In some applications "long" is 200 stack frames, while in others it could be as low as 30. Allow me to rephrase that: 

- **Idle threads** [short call traces] - threads have started up and initialized and now are waiting for tasks to work on; in standard Java web apps I like to take 30 frames as a point of reference and look for threads with call traces ~30 frames long; could be less, could be more. These threads are idle, regardless of their Thread state.
- **Busy threads** [long call traces] - threads have received a task to work on and are currently processing it. While processing a task, their call traces will usually grow significantly larger than idle - could easily be hundreds of frames. These threads are busy, regardless of their Thread state.

Usually, we need to find those busy threads first.

#### There are no busy threads
It may happen that the application is running slow or "hanging" and there are no busy threads. The most likely reason is a bottleneck somewhere outside the application process. I mean, if the application is not doing anything (idling), then it's not slow -- no one is giving it any work to do. Was the request lost or discarded in transit, even before reaching your application process? Is it getting stuck in transit?

The second most likely reason is that the TD was captured at the wrong time: either before or after the slowness took place. This happens more often I'd like to admit. It will almost always happen if you are not the one capturing the trace and you working using someone else's hands ("Ops team, please capture the TD"; 5 minutes later: "here it is"; the issue went away 4 minutes ago), especially if that someone does not know how/when to capture it to obtain **useful** diagnostic information.

The third reason could be that the ST was captured from the wrong process. This is especially the case when requests are hitting a LoadBalancer of any kind (a DNS server is also a primitive LB) and there is no way to determine which server the request will land on. Often Ops take an ST/TD from any random server/instance in the pool that belongs to the application pool and assume that's what you've asked for.

And the fourth reason that I've come across is that you've got the idle/busy threshold wrong. In some applications, processing a task could add very few stack frames, as many as 1 to 5. Look at the traces you have, compare them, and see what's at the top of them. Perhaps you've miscalculated what's a "long" stack trace in your particular case.

#### There are some busy threads
Great, now that you've managed to find busy threads, it's time to see what's in there. Probably the quickest way to see what's happening in a thread is by reading it top-to-bottom - even better - just taking a look at the top 5 or so frames in the call trace. Start from the top. See what the most recent function call was - what's its name? This might hint to you what the thread was doing at the very moment this ST was captured. If the function call doesn't make much sense -- look at the frame below. If still nothing - look below. Keep going downwards until you reach a frame that you can make some sense of. If that frame is a good enough explanation for you of what the thread is doing -- _hey presto_! If not -- try going either back up trying to make sense of what that method you've recognised the frame of is doing inside, or down trying to find what other function has called it.

Once you know what that thread is doing - go to another. Don't worry, it will only take this long for the first few call traces. You'll get up to speed relatively quickly once you get the idea of what to look for and where.

See how many threads are busy doing more or less the same thing. If there is a handful of threads with very similar stack traces - you're probably on to your bottleneck. The higher in the stack you are - the more accurately you can define the nature of the bottleneck. It's best to be able to understand the top 5 frames in the stack.

#### Examples

`java.net.SocketInputStream.socketRead0()` -- means the application itself is waiting for a response over the network from a remote service it depends on. Probably the downstream service is lagging.

---

`org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection()`, that soon leads to a `org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking()` call and parking the thread -- means you have a HttpClient connection pool that's too small to handle the load effectively. You may need to increase that pool and retest.

---

`java.util.concurrent.locks.LockSupport.park()`, when you see some `CompletableFuture.get()` calls lying around - probably you are using a common ForkJoin thread pool for your CompletableFutures, which by default is limited to a very small number of threads (`count(CPU)-1`). Either rewrite the code to use your custom Executor or increase the size of the Common pool.

---


### Flame graphs

#### What they are
While manual analysis of plain-text STs is tedious, difficult and requires a lot of effort, the skill of reading an ST is very useful to gain in order to use more robust ST analysis techniques. My all-time favourite is [FlameGraphs](https://github.com/brendangregg/FlameGraph). At first look, they are funny-looking pictures, kind of diagrams, but very weird ones. Kind of skyscrapers with lots of floors. And the colours don't make it look any prettier.

I agree, flamegraphs aren't pretty pictures. Because that's not what they are. Instead, they are a **very powerful, interactive, multi-dimensional graphical representation of related tree data, like CallTraces**. You don't have to like them to understand them. I like to call them "multiple call traces laid over each other". As if they were transparent sheets of plastic with some similar curves and shapes on them, and you could pile them up and look through the pile and see how different or similar these curves and shapes are.

If all the call traces were identical across all the STs captured, in a flamegraph you'd see a single column with identical stack frames. Simple, because they are all the same, and if you lay them all on top of each other -- they will look the same. No differences.

If we have all traces starting with the same 5 calls and only subsequent calls divided into 2 different sets of function call sequences across all the STs -- in a FlameGraph we'll have a single column 5 frames high, on top of which are 2 narrower columns, representing 2 different sequences of function calls, as stated before.

Now, since STs in busy applications usually have a whole lot of different threads with different sack traces, their flamegraphs will look like a city with skyscrapers.

#### Dimensions

Each FlameGraph has at least 3 dimensions:
- height - that's how deep in the stack the frame is. The higher the frame -- the deeper (the more recent) the call is. Just like in raw StackTraces.
- width - how many such identical sequences of stack frames are, compared to other stack frames
    - if the bar is as wide as the picture -- all these calls have been seen in all the threads
    - if the bar is 50% of the picture width -- 50% of threads had this call in their STs at this same position (with the same sequence of preceding calls); the remaining 50% had different calls at that depth of the stack
- colours - that's optional, though a handy dimension: it highlights calls made by functions written by us, and dims down function calls inside libraries. It's just easier to filter out the calls we're primarily interested in from the ones that libraries are doing internally.
- text - describes what was the function call (basically an excerpt from the CallTrace)
- other - optionally, there might be on-mouseover pop-ups providing even more info on each frame


Each frame in the flamegraph is clickable. By clicking on a frame you can expand/zoom in its tree of sub-calls (i.e. make the clicked frame 100% wide and widen all the upper frames respectively). By clicking on frames below the widened one you can zoom out.

I know, it's a lot to take in. Developers I tried to show flamegraphs to were confused and reluctant to learn using them, thinking it's an overly complex tool, requiring a lot of knowledge/skill/time to get any use out of. IT'S REALLY NOT!!! If you can read CallTraces - reading FlameGraphs will be just as easy, if you bother to play with them for 10 minutes for the first time. You shouldn't need more than 10min to understand the concept.

#### Why bother?
Aaahh, a fine question. And it IS worth the bother because it reduces the amount of data you have to analyze from 70MB of plain-text TDs down to a single image that all fits on your screen. Moreover, a FlameGraph has most of the analysis already done for you, and visualised it!

In general, when I'm analysing FlameGraphs, I'm looking for the largest clusters of identical CallTraces. They will be depicted as skyscrapers with the flattest rooftops. The higher the skyscraper and the flatter the rooftop -- the more likely that's a bottleneck. But how are buildings and CallTraces related..?

- the higher the column - the deeper the thread is in its task execution. Remember, when analysing raw STs we are focusing on busy threads -- the ones with large call traces. 
- the more threads have that same call trace (up to some particular frame), the wider the tower/column will look

#### Example

Consider this Clojure example [link](https://www.janstepien.com/flames/flames.svg):
![Clojure example](https://www.janstepien.com/flames/flames.svg)

Let me break it down for you. We're looking at separate "towers". Here we can see 4 significantly different "towers"
![Flamegraph breakdown into 4 towers](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ccphuvha02kto8okz5hx.png)

1. lots of sharp spikes on this one, meaning several threads are working on this kind of task (the base of the tower is somewhat wide), but while processing those tasks threads are doing all kinds of things (different spikes). There's no 1 thing multiple threads would be stuck on (the same frame across multiple threads would be represented as a wide cell: the more threads - the wider the cell). I don't see any problems there.
2. The roof of this tower is so wide that it's hard to even call it a tower. More like a plane (geological)... The stack depth is definitely more than 30 frames. Are these busy or idle threads? If idle - we don't care much for them, but if busy - ... That's worth looking into. It's hard to imagine an idle thread that would be waiting for new tasks while calling `Object.hashCode()`, so these must be busy threads. Why are soooo many threads (because the column is very wide) stuck at calling `Object.hashCode()`? This definitely looks like a severe performance problem, as normally hash calculation is too quick to even appear in StackTraces. And now we have not one, not two, but over 50% of all the threads stuck there! Does this hashcode function have some custom implementation? Need to look into it.
3. This tower is the narrowest of 4 and the rooftop has lots of antennae and no flat planes for a helicopter to land on. That looks like a healthy execution - don't see any problems there!
4. That's a relatively wide tower, with a wide base. But look at the top - it has a few spikes and a big flat surface at the very top! This flat rooftop suggests that several threads are stuck at an identical function call and cannot proceed further. What's written on that rooftop...? I can't see well. Zoom in on that tower by clicking on one of the widest frames on it. ![Zoomed-in tower 4](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f52m2r4m3xzb5cim7tbf.png) There, all better now. Looks like several threads are stuck calling `Object.clone()`. It could be normal in some cases because `clone()` might have to traverse a large graph of objects (if it's a deep clone), or it might be another performance issue. Might be worth looking into, right after the elephant named `Object.hashCode()` leaves the room.

### Reverse-roots Flame graph
Once you get a good grip on FlameGraphs, I'd like to introduce Reverse-roots Flamegraph (RrFG) to you. Allow me to do that through an example.

Consider a FlameGraph below. The application is sluggish and takes its time to respond to some requests, while it deals with other requests very quickly. The common denominator between the slow requests is that the application is making downstream network calls in order to process those requests. So we suspect network-related issues. Now in this FlameGraph we see several high columns, some with flat rooftops, others - with spikey ones. How do we know which ones are making network calls? Well, we hover a mouse over each and every rooftop and see what the popup says - whether it's a `socketRead0()` or not. And this is a tedious and error-prone job to do. See the FlameGraph below -- the circled rooftops are `socketRead0()` calls. Can you tell how many threads are doing that call? I certainly can't, w/o a notepad, a pencil and a calculator and half an hour to spare. Wouldn't it be nice to have all the similar "roofs" aggregated in one place?

![Normal FlameGraph](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yq4qiqytgg2aee1lrabj.png)

This is where the RrFGs come in. And it does exactly that. It reverses all the StackTraces and draws the FlameGraph as before. However, before reversing, most of the STs started (at the bottom) with `Thread.run()` and ended with `Object.wait()`. After reversing, these STs now start with `Object.wait()` and end with `Thread.run()`. Not only does this manoeuvre change the order of reading a stack, it also now sums up all the `Object.wait()` calls into a single cell at the very bottom. Similarly, it aggregates all the `socketRead0()` calls as well. Now we can see the total number of `socketRead0()` calls, how they compare to other calls across the threads, and we know where these calls originated from (by reading up through the column).

![RrFG](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pmzhgsaaerigut8ji9e6.png)


### Series of STs. Profiling
Analysing a StackTrace gives you plenty of useful information to work with, but that is all assuming you have the right ST. In some cases it might be difficult to obtain an ST reflecting the problem in question, especially when it lasts for up to a second, more so - when it's not easy to reproduce it or its occurrences are unpredictable. So instead of fishing sprats with a harpoon, we can cast a net and hope we catch at least one of them. We can do this so by capturing StackTraces of the same process every 500ms or even more frequently and appending them to a file. This increases the chance that we'll catch the application at the time it's struggling and capture the informative ST we need for analysis.

Suppose you're a micromanager. You have a developer under you and you want to know what he's doing. The easiest approach is to come by his desk every 2 minutes and see for yourself. 
- 09:00-09:28 he's absent
- 09:30-12:44 he's at his desk
- 12:46-13:18 he's absent again
- 13:20-14:52 he's at his desk
- 15:54-15:58 he's absent
- 16:00-18:00 he's at his desk

Seeing the results we can speculate that the developer was 28 minutes late in the morning. Then he was working uninterrupted until 12:46, when he probably took a lunch break. 13:20 he came back and worked continuously to the EOD with a single short break at 15:54 - probably a bathroom break. What happened between each 2minutes poll - we don't know, maybe he was working, maybe running around naked - there's no way to know. But from the sequence of data we have, it seems like he's a pretty good and hard-working employee.

The same thing we can do with a sequence of StackTraces. If we poll the application's stack often enough, we may get a good enough idea of what it's doing over time: where it is fast, where it is slow, and where it idles. The closer the polls are, the better granularity of the application workings we get, and the more accurate application profile can we create. Of course, polling for 8 hours every 500ms or less might be overkill, not to mention the amount of data created. It's a good idea to only profile an application during the period it's having issues. As for the amount of STs to analyze -- either squash all those STs into a single FlameGraph and see at which functions the application was taking most of the time, or create a FlameGraph for each capture of the STs and compare them with each other. Or, you know, group STs by seconds (or 5sec, or so) and generate FlameGraphs out of each of the groups and compare them visually to each other. Or do all of the above.

## Pitfalls
While StackTraces are tremendously useful diagnostic information, we should be careful both capturing and analysing them.

### Capturing
It's very important WHEN the StackTrace is captured. In order to be a useful piece of diagnostic data, it must be captured at the right time -- right when the application is experiencing issues. Even 1 second too late or too soon means a world and rules the ST unfit for analysis.

### Analysing
Having a tool as powerful as FlameGraph you might be tempted to start using phrases like "this method took longer to execute than <...>". There's no "longer" in StackTraces, just as there's no "quicker" **. There is no time duration in StackTraces**, just like there's no timespan in a 20-year-old picture of you as a kid. Merely by looking at a picture I would not be able to tell how long were you standing in that pose. And this is especially the case when squashing a ThreadDump with lots of threads running into a FlameGraph. It didn't take `Object.hashCode()` 50% longer to execute than any other method in the application. The right way to say it would be: "50% of busy threads were seen calling `Object.hashCode()`".  And knowing the swift nature of `hachCode()` we may **speculate** that it's not normal for this method to appear in STs unless it takes longer to execute than expected.

We _may_ have a sense of time if we capture a series of STs and squash them into a FlameGraph, because we do not have the timespan - these STs have been captured over time. But be careful, as the dimension of time is mixed with the dimension of threads-per-TD. Each snapshot you take may have a number of threads that are executing some function calls. And as tasks are executed, they tend to be picked up by different threads to execute. And no one says that execution can't last for 500ms sharp -- just long enough for you not to notice any difference between ST polls. And if you don't notice either thread swaps or execution durations (which you probably won't, as you don't know what happens between ST polls), you'll be tempted to assume a continuous execution of some time duration.

Be careful. StackTraces have nothing to do with the duration of execution. It might suggest it, assuming you capture a sequence of STs with fine enough granularity, but there always is a chance you missed an interruption/change of execution between the polls.


## References
- https://www.ibm.com/docs/en/sdk-java-technology/7.1?topic=ij-threads-stack-trace-threads-1
- https://unix.stackexchange.com/questions/166541/how-to-know-where-a-program-is-stuck-in-linux


> Written with [StackEdit](https://stackedit.io/).