I agree with spawning concurrent workers, but how do you keep those 10,000 jobs from exhausting the db connection pool? I started wrapping my job logic with a .with_connection block to keep the job from using 10,000 connections to the database.
How do you execute both concurrency and database connection handling?
For our team, we make the connection pool size = concurrency so we don't have to think about the concurrency exhausting the DB connection pool. With this, we have to trade-off having high levels of concurrency for being sure that our DB connection pool won't get exhausted.
I agree with spawning concurrent workers, but how do you keep those 10,000 jobs from exhausting the db connection pool? I started wrapping my job logic with a
.with_connectionblock to keep the job from using 10,000 connections to the database.How do you execute both concurrency and database connection handling?
For our team, we make the connection pool size = concurrency so we don't have to think about the concurrency exhausting the DB connection pool. With this, we have to trade-off having high levels of concurrency for being sure that our DB connection pool won't get exhausted.
There's a whole debate on the Sidekiq repository about this: github.com/mperham/sidekiq/issues/...