DEV Community

Tushar  Borole
Tushar Borole

Posted on

Scheduler Design Help Needed

Basically I am designing a system, which can have thousand of cron jobs. I am creating a UI where one can create cron job, and specify the interval in it. What system design you will suggest making it scalable?

More Details: My Question is in terms of the backend infrastructure. So I am going to use the package called npmjs.com/package/html-differ to compare two HTML. There will be a user interface where the user will define the webpage URL and interval time where webpage changes will be checked. Let's assume we have 10k URL to check with different time intervals. I wanted to have a recommendation about the database or queuing system like rabbitmq i should use to make it scalable and fault tolerance.

Top comments (6)

Collapse
 
devdrake0 profile image
Si

What have you thought of so far?

Are you asking for advice on what technology stack to use, or just a specific part of it (e.g. DB)?

What do you have experience in?

Collapse
 
tusharborole profile image
Tushar Borole

I am a full-stack engineer, My Question is in terms of the backend infrastructure. So I am going to use the package called npmjs.com/package/html-differ to compare two HTML. There will be a user interface where the user will define the webpage URL and interval time where webpage changes will be checked. Let's assume we have 10k URL to check with different time intervals. I wanted to have a recommendation about the database or queuing system like rabbitmq i should use to make it scalable and fault tolerance.

Collapse
 
devdrake0 profile image
Si

You're asking an architectural question, which isn't easily answered without full context of what you're trying to make, what technologies/languages you want you use, or how much your budget is.

I'd also suggest text based communication isn't the right place for such an architectural design conversation.

Thread Thread
 
tusharborole profile image
Tushar Borole

I am creating an open-source project
Thanks for being so responsive and helping on this

Thread Thread
 
devdrake0 profile image
Si

Let me have a think of some questions that will help me understand your needs more, and maybe we can figure something out together πŸ‘

Thread Thread
 
tusharborole profile image
Tushar Borole

Awesome, I havenot did any commit on this project, but if it do, it will be here github.com/tushariscoolster/Telesto