Recently, I tried using a stream-based CSV formatter in a browser-based (actually ElectronJS) application. It was ridiculously slow, processing less than 100 records per second. These rows weren’t crazy long or anything, either. The problem can be traced back to the usage of Streams in the browser, though there are a couple methods that can be used to speed the process up. I suspect that this affects other applications as well.
The root of the issue can be traced back to nextTick(), a method which should wait just long enough for other code to run in NodeJS. When ported to the browser, this becomes setTimeout(), which must wait for the next animation frame.
So when is nextTick() fired? After every single chunk is read. This really grinds things to a halt when nextTick() only fires after 1/60th of a second. The simplest solution is to run the transform directly and synchronously, bypassing the streams entirely. If you’re dependent on some upstream library that uses streams, I’d recommend patching it to expose the transform function directly.
In my case, simply re-writing the transform was sufficient, and sped up the procedure from multiple minutes to less than a second.
Top comments (0)