A Logger : What to consider when creating a Logger with Node.js

Lately at work, we were facing an issue that the systems that were interacting with each other have grown into more complicated systems, it was getting difficult and more difficult to track the flow between them.

we needed more visibility of these systems which was planned to help us in areas such as :

1- Easy to Debug Issues when something fails
2- Easy to identify issues that are occurring in the run time.
3- Reducing Debugging time

So that's how we have decided on a Logger, We needed a General and Centralized Logger that we could add more visibility.

What can you log?

To be honest you can log as much as pieces of information you think are possible to have.
the way you decide for creating your own logger is to store what you need, in our case :

1 - we are keeping track of a record that was changing between multiple systems, status changes of this record and we are storing it.

2- we are storing the execution of the code, and the parameters changing between systems.

3- we are doing some heavy calculations and storing them so it will be easier to get more complicated reports from this logger

in your case :
You can track your users across your systems, track requests, Track System behavior, Track System activities, and much more.

A Logger can help you in your business decisions, you can use them in BI tools or any data science procedures.

However, there are some things to consider while creating a logger in Node.js.
You may store too much information and too much interaction with I/O with any device or with any database.

1 - Don't Block your Request-Response Cycle and Main Thread ( Node.js )

A ) Don't "Await" each one of your logger inserts, you know it will stop at each line to get success then next line?
You can parallelize them with Promise.all() and use Single await for them, the tasks will run in parallel.

Await Promise.all([promise1, promise2, promise3])

B) Let's make this better, do you know this won't get to the line until they all run in parallel and all of them succeed?

Node.js is single-threaded but it will launch other threads in the background so that they run without blocking your code, HMMMMM let's make turn this to benefit our use case but how?
in Node.js, we have to Await as well as .then to resolve promises ... so simply instead of Await and waiting for a task to complete, just add a .then() to it so that it will resolve, The Cool part is that the code execution will continue and will execute other things and this specific task runs in the background.

Promise.all([promise1, promise2, promise3]).then((values) => {
  console.log(values);
});

C ) That's really cool, now you have a nonblocking logger that runs in the background, but still the request won't end because there is something in the background that is executing, and it's taking too much time.

There is a solution for that, You need to run your loggers after the response has been sent back to the client. in Node.js, the code will continue executing even after ending the request which is basically what we want, to not make the users wait.

res.send('Hello World!')

//DO SOMETHING HERE 

2- If you are storing in the database, Don't overwhelm the DB with too much inserting, Best practice is to use batch Insert or batch Update.

Or You can use Queues to reduce the load and reduce the Database CPU Usage ( if it easily hits the maximum )

Example: AWS SQS service

3- Don't overwhelm your Server with too Many HTTP Requests

if you are storing a user location, it's a bad idea to track the user for each second and let the server know to store a logger.
You can hit the Server for example per 20 seconds, or you can open other connections that HTTP like Socket Channel to send the logs and reduce the load.

4- Try To save meaningful data and try to think in the long term about where you may use these logs in a more effective way.

in the conclusion, This is how I approached this problem of Logging the things I want in an effective way that its Logging ( First Priority ) and its Not slowing down anything ( Second Priority )