In this post, I show how to implement log rotation in a Python project.
What's log rotation?
From Wikipedia:
In information technology, log rotation is an automated process used in system administration in which log files are compressed, moved (archived), renamed or deleted once they are too old or too big (there can be other metrics that can apply here). New incoming log data is directed into a new, fresh file (at the same location)
So, it's important to consider log rotation specially in that systems that produce a great quantity of log messages.
There are different criteria of log rotation and in this post I'll focus on timed log rotation and file size rotation.
How we implement logging in Python?
Super easy, just using the logging module.
The logging module offers a default logger named "root". This logger allows you to get started with no configuration.
import logging
logging.info('This is an info message')
logging.warning('This is a warning message')
logging.error('This is an error message')
logging.critical('This is a critical message')
This module permit a lot of customizations like:
- Logging severity level
- File name
- Log format
- etc.
You can create a custom logger using the class Logger, defining a basic configuration (date format, log format, etc.).
logging.basicConfig(
format='%(asctime)s %(levelname)s %(message)s',
level=logging.DEBUG,
datefmt='%m/%d/%Y %I:%M:%S %p',
)
# Log formatter definition
log_formatter = logging.Formatter(fmt=' %(asctime)s %(levelname)s %(message)s')
# Getting the logger
logger = logging.getLogger('my-logger')
# Using the logger
logging.debug('This is a debug message')
logging.info('This is an info message')
logging.warning('This is a warning message')
logging.error('This is an error message')
logging.critical('This is a critical message')
To implement different ways to log information, Python uses log handlers.
The log handlers are components that effectively writes logs, and they give you the opportunity of:
- Display logs in the console (StreamHandler)
- Write logs on files (FileHandler)
- Write logs on files with log rotation based on file size criteria (RotatingFileHandler)
- Write logs on files with log rotation based on time conditions (TimedRotatingFileHandler)
- Sending logs via SMTP (SMTPHandler)
- etc.
You can create your custom handler or extend a pre-existing handler.
File Logging with Python
To implement file logging, we'll use the FileHandler, a handler able to write logs to the file system.
Below is shown how to define this handler using the same basic configuration defined previously.
# File handler definition
fh: logging.FileHandler = logging.FileHandler(
filename=f'./logs/my-app-file.log',
mode='a'
)
# Setting the formatter for the handler
fh.setFormatter(log_formatter)
# Setting the severity level for the handler
fh.setLevel(logging.INFO)
# Adding handler to the logger
logger.addHandler(fh)
#Trying the handler using the handler
logger.info('This log is an info message')
This handler simply writes log messages to a file named my-app-file.log.
Log rotation with Python
File size log rotation
To implement logging rotation based on the size of the file, we'll use the RotatingFileHandler. This handler is used to write logs on the file system and to trigger a rotation when a specific size of the file is reached.
Below is shown how to define this handler using the same basic configuration defined previously.
# Rotation file handler definition
rfh: RotatingFileHandler = RotatingFileHandler (
filename=f'./logs/my-app.log',
maxBytes=100,
backupCount=5
)
# Setting the formatter for the handler
rfh.setFormatter(log_formatter)
# Setting the severity level for the handler
rfh.setLevel(logging.INFO)
# Adding handler to the logger
logger.addHandler(rfh)
# Trying the handler using the handler
logger.info('This log is an info message')
In this example, we've set a max dimension of 100 bytes and a backup count of 5 log files (this means that we keep at most 5 files from the log history).
A result of this rotation after writing more than 1024 bytes could be something like shown below:
my-app.log
my-app.log.1
my-app.log.2
my-app.log.3
my-app.log.4
my-app.log.5
Timed log rotation with Python
To implement file rotation logging based on time event, we'll use the TimedRotatingFileHandler that is the handler used to write logs on the file system and trigger a rotation when a time condition is satisfied.
Below is shown how to define this handler using the same basic configuration of the previous examples.
rth: TimedRotatingFileHandler = TimedRotatingFileHandler(
filename=f'./logs/{log_file_prefix}.log',
backupCount=5,
when='midnight',
)
rth.setFormatter(log_formatter)
rth.setLevel(logging.INFO)
rth.namer = lambda name: name.replace(".log", "") + ".log"
logger.addHandler(rth)
logger.info('This log is an info message')
In this example, we've set a daily rotation triggered each midnight and a backup count of 5 files.
A result of this rotation after 5 days could be something like shown below:
my-app.log
my-app.2023-10-12.log
my-app.2023-10-11.log
my-app.2023-10-10.log
my-app.2023-10-09.log
my-app.2023-10-08.log
For this handler, we've defined a namer with a lambda function that has the goal to preserve the extension of the log files. This namer get in on the action when a rotation happens.
Custom File timed log rotation with Python
It's possible to create the own log handler extending a pre-existing handler. For instance, we can extend the TimedRotatingFileHandler class in a way to introduce, for each log message emission, a check about the presence of the file. In this way, if someone or something deletes the file, the handler is able to create it again, and we don't miss further log messages.
Let's create the class:
class TimedRotatingFileHandlerWithFileCheck(TimedRotatingFileHandler):
"""
Handler for logging to a file, rotating the log file at certain timed
intervals. This handler based on TimedRotatingFileHandler offers a file
presence check for each log emission.
"""
def __init__(self, filename, when='h', interval=1, backupCount=0,
encoding=None, delay=False, utc=False, atTime=None,
errors=None):
super().__init__(filename, when, interval, backupCount, encoding, delay, utc, atTime, errors)
def emit(self, record):
if not os.path.isfile(self.baseFilename):
os.mknod(self.baseFilename)
super(TimedRotatingFileHandlerWithFileCheck, self).emit(record)
In this version, before to emit a log, we check the presence of the file.
Conclusions
In Python, logging is super easy! The logging module offers a lot of possibilities to customize and implement the strategy more suitable for you. The presence of a lot of handlers and the possibility to have a high level of customization makes this module very useful.
In my opinion, having a good and balanced logging system is a must for each system or application.
Suggestions and corrections are welcome.
Top comments (0)