Integrate logging into your C++ Project. Do it right — do it once!

Almost half of C++ Linux projects I've been working with always had some kind of logging implemented, but it never had any set of rules or practices discussed before starting the project regarding when logs should be added, what should be message format, how many log levels is needed for this exact project, where logs should be stored, how it will be rotated/accessed, etc. As a result nobody paid attention to adding meaningful/accurate logs, and nobody could rely on getting useful insights on "why your server crashed last night" from logs.

Why logging is important?

Because sooner or later you will run your C++ server on some remote machine, and you will need to monitor how it runs, or investigate when it crashes ASAP. With good structured and easy readable logs you should be able to do it really ASAP.

Basically, there always were two kind of projects: some which are just starting, and some which were started few years ago, and by now have pretty solid number of code lines. If you are dealing with the second one your are in the worst position - integrating logging into existing big codebase may be painful, and even, if you have it integrated already, but nobody guarded it this gonna be long and boring story changing it now.
Let's consider the case when you and your team are just starting new C++ project from scratch. This is just right time to integrate logging and agree on logging rules before it's too late and not a priority. Let's do it asking right questions!

3 questions to be answered before start adding logs

1. What logging library is best fit for my C++ Linux project?

There are few options of C++ logging libraries. Which one to use depends on how big and complex your project gonna become. But besides that the minimal set of features it should has is:

• multithreading/async writing to log files,
• configuration from config file and trace filtering from config,
• log rotation support,
• good documentation and library support itself. There are around 10 possible options for C++: Pantheios, Glog, log4cpp, P7, G3log, Spdlog, Easylogging, Boost.Log. So, what library to pick? If your project is kind of small, hobby projects pick Glog or Log4cpp. Otherwise, you should be looking for well optimised and customasible solution which could be P7 or boost::log. Interesting fact: among all libs, only Glog has crash handling support (SIGSEGV, SIGILL, SIGFPE, SIGABRT, SIGBUS, SIGTERM). Which is really good for C++ aplications.

2. How are you gonna read your logs? 

This may affect logs format and where you keep your logs. Here is an article where you can read about 4 different tools which let you read logs in Unix/Linux systems: Best Linux Log Monitoring and Management Tools.
If you picked P7 library, it will be useful to take a look at theirs own tools: Baical.

3. What should be logs message format?

Kind of standart format is: [Time][Log level] - [File]:[Line] [Function] - [Text]. If app is multithreaded/modular, add [Module][Thread] options as well. Set names to threads, don't be lasy, it will be much more readable.
Above you can see an example how logs could be visualised if you set up it right.

Logging is not easy. It requires special way of thinking and balancing. At some point you will naturally run into question: what severity level should this log be, and how many logs should we add to not make code messy?
There is no exact instruction on this. You should make up you own pattern. Here are readings I recommend on this topic: Logging Level, Logging Best Practices. 
In short, the way you should be thinking about it is: "what info will be potentially helpful if something goes wrong in this exact function?". Really think about it. Also, "will somebody who didn't develop the code understand log message?". In other words: "Is there enough context?". 
When you fix some new bug or review someone else bug fix, also, think in this way: "what log info would be helpful to find this bug sooner?", and don't be lazy to add it!