Map is fantastic, but I like being able to see progress of my tasks correctly using tqdm, for that I use the following pattern:
from concurrent.futures import as_completed
output = []
with ThreadPoolExecutor(num_threads) as ex:
futures = [ex.submit(lambda (ii, query): (ii, query_func(query))) for ii, ean in enumerate(queries)]
for f in tqdm(as_completed(futures)):
output.append(f.result())
output = [s[0] for s in sorted(output, key=lambda x: x[0])]
Yeah, there's a lot of neat stuff in that module. I chose map because it's probably the quick and dirty solution to make a sequential workload into a parallel one.
I have a program in production that does a mass upload of images to S3 using futures and as_completed.
Didn't know about tqdm though, thanks for the tip!
Map is fantastic, but I like being able to see progress of my tasks correctly using tqdm, for that I use the following pattern:
Yeah, there's a lot of neat stuff in that module. I chose map because it's probably the quick and dirty solution to make a sequential workload into a parallel one.
I have a program in production that does a mass upload of images to S3 using futures and
as_completed.Didn't know about tqdm though, thanks for the tip!
tqdm is a life changer!