Fast folder iteration in python

Post created at 2022-01-19 07:53

I have a problem to deal here.

I need to iterate over a big tree of folders and files and run process over each file.

In python, we have some options to deal to folders and files.

• glob
• iglob
• os.walk
• os.scandir
• pathlib.Path

Running this benchmark, we can read some implementation details:

glob

import glob

folder='some_folder/another_folder/**/*'
for file in glob.glob(folder, recursive=True):
 print(file)

some_folder/another_folder/f1/file1
some_folder/another_folder/f1/file1
some_folder/another_folder/f1/file2

Pros

• Easy to use, less code to write
• Easy to apply filters using masks (fnmatch based)
• Returns data as a list

Cons

• Time to scan is the worst (more than 11x the quickest method)
• Data will be available only after all files/folders been scanned

iglob

import glob

folder='some_folder/another_folder/**/*'
for file in glob.iglob(folder, recursive=True):
 print(file)

some_folder/another_folder/f1/file1
some_folder/another_folder/f1/file1
some_folder/another_folder/f1/file2

Pros

• Easy to use, like glob
• Same filtering
• Returns data as iterator, so you can have your data without need to wait all iteration ending.

Cons

• Time to scan is a little better than glob (10x the quickest method)
• Time to first file could be mutch better

os.walk

import os

folder='some_folder/another_folder/**/*'

for root, _, files in os.walk(folder):
 for file in files:
 print(os.path.join(root, file))

Pros

• Good performance. Second quickest method (less than 2x slower than the quickest)
• Time for the first file almost imediattely (better result of all)
• Explicit code in loops gives more visibility and control if you need validations or another nasty processes

Cons

• More code needed, with nested loops.
• If you need to nest os.walk in another os.walk loop, some strange things can ocurr. But probably your code needs some refactoring.

os.scandir

import os
from typing import Generator

folder='some_folder/another_folder/**/*'

def get_files(folder: str) -> Generator:
 with os.scandir(folder) as scan:
 for item in scan:
 if item.is_file():
 yield item.path
 else:
 for subitem in get_files(item.path):
 yield subitem

Pros

• Best performance of all
• Context based, assure resources are released after processing
• Less verbose than os.walk
• Easy to implement your custom business rules

Cons

• A little more complex. No big deal.

pathlib.Path.rglob

import pathlib

folder='some_folder/another_folder'

for path in pathlib.Path(folder).rglob('*'):
 if path.is_file():
 yield str(path)

Pros

• Easy to implement your custom business rules
• Returns a iterator
• You can have your files soon and don't need to wait for all scan (7x more time than os.scandir)

Cons

• Average time to scan (almost 8 times greater than os.scandir)
• Memory consumption is 4 times more than the other methods

Some data from tests

System Information

CPU Info

CPU Usage Per Core

Memory Information

SWAP

 Creating sample files
 + Creating folder ./test_files
 + Creating files LEVELS=6 FOLDER_COUNT=6 FILE_COUNT_BY_FOLDER=20
 + Created 933120 files in 0:02:08.928680

* Running iterators: GlobFolderIterator IGlobFolderIterator OSWalkFolderIterator ScanDirIterator PathLibFolderIterator

MEMORY USAGE

ELAPSED TIME

TIME FOR FIRST FILE

You can check the source code for this here.

Image from this wikipedia article.