DEV Community

Discussion on: Introduction to web scraping with Node.js

Collapse
 
aurelkurtula profile image
aurel kurtula

Yes, you could set a timer when you want the scrapping to happen. For example if I wanted to scrape my articled I have it run Thursday and Sunday. If I was scraping twitter I'd have it run every 15 minutes (running it every second would be costly).

If I remember correctly this is how ifttt tasks work. In fact I know because I tried it years back, they don't update when the new content is published but every how ever many minutes/hours.