DEV Community

loading...
Play Button Pause Button

CLUSTER MULTI THREADED NODEJS -- Run threads in parallel to speed up long processes

JavaScript Joel
Cofounded Host Collective (DiscountASP.net). Cofounded Player Axis (Social Gaming). Computer Scientist and Technology Evangelist with 20+ years of experience with JavaScript!
・1 min read

My JavaScript that creates thumbnails for hundreds of files is taking too long. So I have decided to speed things up with cluster.fork() that is built into Node.js. This will allow the workload to be spread out across my multi-core processor system.

8:53 should read: index % forks === cluster.worker.id - 1

To follow along, you can fork this repository. I have added a tag you can check out that will bring you right to the point this video was created.

git checkout https://github.com/joelnet/mojo-gallery.git
cd mojo-gallery
git checkout youtube-video-cluster

Be sure to subscribe for more videos like this!

Alt Text

Discussion (5)

Collapse
omkaragrawal profile image
OMKAR AGRAWAL

I had a similar requirement, for image manipulation, I used workerpool library. It made task pretty easy. Initially I did achieve it with worker_threads , but the issue I faced was that I couldn't actually keep a track of threads, and would quickly run out of cpus, but workerpool managed it automatically.

Collapse
joelnet profile image
JavaScript Joel Author

Ahh! this is a cool library. I haven't seen this one before. Definitely helps with the managing of threads. I'll check this out next time I need something multi-threaded. Thanks for sharing!

Collapse
jorge_rockr profile image
Jorge Ramón

What about using Working Threads? Wouldn't be lighter and faster?

Collapse
joelnet profile image
JavaScript Joel Author

I think you are right. I am familiar with cluster because I use with with Express.js. And I think cluster might be the better use-case when using Express. But in this instance because there isn't much communication back and forth between processes, worker_threads would probably run faster. I think the difference would be small though.

Want to convert it and run some benchmarks? 😁

Collapse
mumingazi profile image
Mumin Gazi

5 at most