Struggling with micro-optimizations on large scale data processing

Roelof Jan Elsinga on October 02, 2019

Struggling with micro-optimizations on large scale data processing Recently I worked on a problem that didn't seem solvable. It was a ... [Read Full]
markdown guide
 

The calculations are CPU bound and hence you can try parallelism (Data Parallelism) by distributing your workloads across multiple cores of your server CPU. You can fork them if there are dependencies. If you have C#, Java stack these languages support that

 

I use PHP for this, but there is a way to distribute processes across multiple threads. I'm going to look into this, thank you for your reply! :)

 

I'd say that it might be worth looking into other languages that might be better at this specific scenario than PHP.

I don't know exactly what your code does and what it's based on, because obviously that idea might not be feasible if you're tied to a framework of some kind to get the data from.

code of conduct - report abuse