Python resource limitation

It turns out that you can explicitly specify the number of resources for a particular application in Python. By resources, I mean CPU, memory, number of processes, number of open files, call stack, etc. Python's resource module in the standard library gives you an easy way to do that and more.

An example of CPU time limitation:

import signal
import resource


def time_exceeded(signo, frame):
    raise SystemExit('Time\'s up!')

def set_max_runtime(seconds):
    _, hard = resource.getrlimit(resource.RLIMIT_CPU)
    resource.setrlimit(resource.RLIMIT_CPU, (seconds, hard))
    signal.signal(signal.SIGXCPU, time_exceeded)

if __name__ == '__main__':
    set_max_runtime(15)
    while True:
        pass

Result:

Time's up!

In this program resource.setrlimit() is used to set a soft and hard limit on a certain resource.

A soft limit is a current limit, which can be reduced or increased by the process over time. When an operating system reaches the soft limit, it usually limits or notifies the process by a signal.
The hard limit is the upper limit of the values that can be used for the soft limit. It can only be lowered to the soft limit and can never be raised by the user process. To raise the hard limit, you must be a superuser.

Example of limiting the number of processes:

import signal
import resource
import multiprocessing

def set_max_processes(n):
    _, hard = resource.getrlimit(resource.RLIMIT_NPROC)
    resource.setrlimit(resource.RLIMIT_NPROC, (n, hard))

if __name__ == '__main__':
    set_max_processes(2)
    def f(x):
        return x*x

    p = multiprocessing.Pool(5)
    print(p.map(f, [1, 2, 3]))

Result:

BlockingIOError: [Errno 11] Resource temporarily unavailable

It would be best to check the documentation page of the resource module to find out what other resources can be restricted.

Thank you for reading!

Any questions? Leave your comment below to start fantastic discussions!

Check out my blog or come to say hi 👋 on Twitter or subscribe to my telegram channel.
Plan your best!