I had to place thousands of messages on a queue and collect each response by an ID. This is how I used multiprocessing to do that, so that messages were processed in parallel to make it complete faster:
//lists to keep the thread instances
List threads1 = []
List threads2 = []
//first task (e.g. putting on Q)
def doThis = { i, j ->
sleep(randNum(100))
println "doThis(${i}) thread: ${j}"
}
//second task (e.g. getting from Q)
def doThat = { i, j ->
sleep(randNum(20000))
println "doThat(${i}) thread: ${j}"
}
//call first task (e.g. for eachRow from DB, put msg on Q)
20.times { i ->
def t1
t1 = Thread.start { doThis(i, t1.name) }
threads1.add(t1)
}
//wait for all task 1 threads to end
for (thread in threads1) {
thread.join()
println "doThis joined: ${thread.name}"
}
//call second task (e.g. for each correlation ID, get msg from Q)
20.times { i ->
def t2
t2 = Thread.start { doThat(i, t2.name) }
threads2.add(t2)
}
//wait for all task 2 threads to end
for (thread in threads2) {
thread.join()
println "doThat joined: ${thread.name}"
}
println "All threads done"
def randNum(maxSize) { //create random wait times
return Math.abs(new Random().nextInt() % maxSize) + 1
}
Top comments (1)
take a look at the groovy gpars library