DEV Community

Discussion on: How I Wrote the Fastest Directory Crawler Ever

Collapse
 
thecodrr profile image
Abdullah Atta • Edited

Hey, thanks for the info. I will definitely investigate and see why fdir is missing files. When I tested I confirmed fdir with my system's file manager's file count (KDE's Dolphin) but I will use some other tool to confirm. Thanks again.

Edit: KDE's Dolphin is giving file count as: 2,407 files (in node_modules) while the command you gave is: 1973. So... either you command is flawed or something else is up.

Collapse
 
tunnckocore profile image
Charlike Mike Reagent

Okay, I see why it was 4700, it was doubled.

Updated the command ls -al node_modules/**/* | grep "^-" | grep -v "node_modules" | wc -l

It basically is 2398, including the dot files, that's closer to your report. Soo.. I don't know.

Thread Thread
 
thecodrr profile image
Abdullah Atta

Yes I tried various other commands. Fdir comes really close and since it cannot handle symlinks (yet) the difference maybe because of that. But overall, fdir wins. 😁😁😁

Collapse
 
tunnckocore profile image
Charlike Mike Reagent • Edited

I don't know. Mine (pcmanfm, on archlinux) is showing 2680, while the command 4716 - git clone, yarn, and then the command.

I more believe on the ls command. The ls -l node_modules/**/* returns all files AND directories. But the grep is getting only the files.

node_modules/urix:
total 20
drwxr-xr-x 2 charlike users 4096 Mar 20 08:38 test/
-rw-r--r-- 1 charlike users  308 Jan  6 23:04 index.js
-rw-r--r-- 1 charlike users 1079 Jan  6 23:04 LICENSE
-rw-r--r-- 1 charlike users  494 Jan  6 23:04 package.json
-rw-r--r-- 1 charlike users  812 Jan  6 23:04 readme.md

So above will give us 4 here, instead of 5.

edit: In anyway good job. The difference of 2600 to 2900 is most probably because they (rrdir for example and others) include the dirs in the final results array. Both fast-glob and fdir report same count, because both don't include dirs. So, don't think that one too much. The interesting thing is why only ~2k are reported and not 4k which is more likely the real number of files.