The web crawler is an efficient way to get the data if we don’t have REST APIs or libraries to retrieved data.
What we the most want to do with web crawlers is retrieving data in real-time. In a program of a web crawler, it usually sends a request to the target website as a flight company, EC website, or galleries of products. Then parse the response from the website and extract the information we expect.
This article introduces the following topics.
- Implement a web crawler program for static websites
Please refer to my article
There are some of my articles and released projects. Feel free to check if you like!