You could use multiple processes, but I suggest a simpler solution, boost your thread pool size and use parallel scheduling and scheduling queues to schedule more than one task just remember to keep the number of spawned tasks under the number of threads in the thread pool.
How’s it going, I'm a Adam, a Full-Stack Engineer, actively searching for work. I'm all about JavaScript. And Frontend but don't let that fool you - I've also got some serious Backend skills.
Location
City of Bath, UK 🇬🇧
Education
10 plus years* active enterprise development experience and a Fine art degree 🎨
It's a shame I didn't see this comment, I went the fork process route (1 per jar), it's now 300 processes for allocated 'jobs'
Parse jars
bytecode forEach classpath
Parse bytecode to typescript
Although it's not sequential, some jobs finish sooner than others and then when the child proc does we pass on to the next one
The multi process is a hell of a lot faster but now running kind of stalls and lags my system... One of the jobs is the classpaths processing, that could have quite a lot of org.paths.goo paths so I think instead of multiprocess here, maybe a queue and limit the amount of processing within that job?
Your suggestions I will need to research 🦉
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
You could use multiple processes, but I suggest a simpler solution, boost your thread pool size and use parallel scheduling and scheduling queues to schedule more than one task just remember to keep the number of spawned tasks under the number of threads in the thread pool.
It's a shame I didn't see this comment, I went the fork process route (1 per jar), it's now 300 processes for allocated 'jobs'
Although it's not sequential, some jobs finish sooner than others and then when the child proc does we pass on to the next one
The multi process is a hell of a lot faster but now running kind of stalls and lags my system... One of the jobs is the classpaths processing, that could have quite a lot of
org.paths.goopaths so I think instead of multiprocess here, maybe a queue and limit the amount of processing within that job?Your suggestions I will need to research 🦉