This post is originally published on Quora, with code links at the end
I used 58GB of RAM once. NodeJS refuses to use more memory and I had to rewrite the program in C++.
It's in a research project. I captured a packet trace from Network File System (NFS) server, and I want to reconstruct the complete pathnames accessed in each command. The way NFS works is that, every command only carries a component of the name (like a directory name or a filename within a directory), along with a handle that represents the directory in which the file resides. To get the complete pathname, I need a lookup table of all known handles and their corresponding pathnames. Then, for each new command that has a handle and one more name component, I can query this lookup table to find out the pathname of that handle, and append the new component to get the complete pathname. Finally, I'd insert the new pathname along with the new handle into the table.
I have 24 hours of packet trace with millions of these handles. It shouldn't take so much memory, but I somehow decided to write the program in NodeJS. NodeJS struggled to allocate more and more memory as it reads the input, until it reaches 58GB after several hours. The machine has 96GB but NodeJS won't use more memory. I can tell that it's no longer making progress, because records in /proc/*/fdinfo indicates that the cursor in input file isn't moving anymore.
I spent four hours rewriting this part of the program in C++. More specifically, I hosted the lookup table in a C++ program, and had the NodeJS program communicate with C++ process via Unix socket. The program completed in 15 minutes with no more than 1GB of memory.
Conclusion: if you need 64GB of memory just for data structures, you probably need a better programming language.
source code links
https://github.com/yoursunny/nfsdump
pathtree/fullpath.js is the Node.js edition, for Node.js v0.6 on Ubuntu 12.04.
pathtree/fullpath.cc is the C++ edition implementing same functionality.
highlighted Quora comments
Michael Johnson-Moore
Conclusion 2: If you need something to be fast, don't use an interpreted language.
Alex Bujorianu
Many more developers should heed this lesson! ... I really don’t understand why a cloud desktop client, a text editor (!) etc. need 100MB of memory per instance.
Ubeyde Mavus
Or you need to know your way around C++ better than your way around NodeJS.
Latest comments (3)
Rendering Crysis main menu.
Interesting! Is there a reason why you did not start saving to filesystem/db, apart from speed? That's an insane amount of memory!!
It's my first non-trivial Node.js program and I was clueless on databases. Moreover, I used to avoid databases because I thought they would fail mysteriously and cause data loss.