I ended up speeding this up by reading the entire file at once instead of line by line. I think the issue was my program was constantly going back and forth with the disk, instead of streaming all of the information it needs at once.
Fortunately for me, the files I’m scanning are small, so it was ok to keep them in RAM for my processing.
I ended up speeding this up by reading the entire file at once instead of line by line. I think the issue was my program was constantly going back and forth with the disk, instead of streaming all of the information it needs at once.
Fortunately for me, the files I’m scanning are small, so it was ok to keep them in RAM for my processing.
Indeed, you would want to minimize the i/o between your program and the file as those are costly in memory. Good that you found a solution! :)