This article originally appeared on IG's blog
So you just wrote another try/catch block and again you start wondering: should I log this exception as an ERROR or a WARN? Or perhaps you wrote another one of those conditional branches that “should never happen” and again you start wondering: what log level should I use for this “impossible” case?
You are asking the wrong question.
These questions are at the wrong abstraction level as they do not help you to come up with an answer.​ Instead you have to look at this question from the perspective of how the log statement will be used.
At IG we have generally come to the following log level conventions:
- DEBUG: information that is useful during development. Usually very chatty, and will not show in production.
- INFO: information you will need to debug production issues.
- WARN: someone in the team will have to investigate what happened, but it can wait until tomorrow.
- ERROR: Oh-oh, call the fireman! This needs to be investigated now!
With this usage in mind, the log level question becomes: “Do I want to wake up in the middle of the night to investigate this, or can it wait until tomorrow morning?”, which is very concrete and a question that even your Business Manager/Production Owner should be able to answer.
Of course this convention needs to be taught to new team members, and constantly reminded to existing ones. And as much as we love wikis, code reviews and email reminders, having to do the mental mapping from the log level to how the log statement is going to be used is not a natural one, and it is going to be forgotten.
So, after one of those code reviews where I asked several times “Why do you want to wake me up in the middle of the night for this exception?” I decided that we would switch the logger interface from SLF4J to an experimental “IGLogger” one which looks like this:
The results were interesting:
- Laughs when reading “wakeMeInTheMiddleOfTheNight”
- Extra care when using “wakeMeInTheMiddleOfTheNight”
- Less mental gymnastics when logging
- Way easier to make the correct logging choice
- Nothing to explain to new team members, nothing to remind to the existing ones
- Adopted by developers’ choice on the rest of the team’s projects
And last, but not least, we got more sleep: the number of ERROR logs was reduced to 0, not because nobody could be bothered to type “wakeMeInTheMiddleOfTheNight”, but because this service was an offline, non-critical one, so any issue could wait for the next day.
Log levels are the wrong level of abstraction.
Top comments (4)
It's good to have concrete definitions of what the error levels mean. It's also nice to eliminate as much as possible from live logs.
On a previous project we defined "Error" the same way, but we defined "Warning" a bit differently. We said that warnings are conditions are conditions that if they occur sometimes are okay, but if start appearing frequently, or too many at a time, are a problem.
We used those to mark code paths that were inefficient, or fallbacks to preferred algorithms. Something we'd expect to come up a couple of a times of day due to the operating environment. If they happened often it meant they were causing a significant problem for the tool.
Thanks for the comments.
We have toyed with the idea of adding a "wakeMeIfThisHappensTooOften" method to the logger to cater for that use case. The hard question is what means "too often".
In practice we have found that most times the places have this use case are protected by a circuit breaker so monitoring the circuit breaker is enough.
I would avoid having that couple of alerts every day if they are systematically ignored. One day they are two, next day they are four, ... One of the last projects that I worked at ended up with 13000 of those alerts which prompted me to write this blog post
As someone who has often wondered which one to use, I like the spin this puts on logging and it makes a lot of sense. The old logging method names don't really convey intent like we've been taught to do for so long. I dig this. Thanks for sharing.
Glad you liked it. You will probably find Steve Freeman Logging is also a feature interesting.