how would you achieve the following logic using python?
- Take a search query, for example, why do I like dogs?
- Open browser, navigate to duckduckgo (or something else), search for my query.
- Save the HTML of the search page.
- Open each URL in search page (for the first page)in a new tab.
- Save the HTML of each opened URL
Top comments (2)
Why do you need to open the pages in the browser? Wouldn't it be easier to just download the HTML?
https://duckduckgo.com/?q=dogs
with requestsThis is the simplest version I can think of. There are other ways to scrape pages and links.
If you truly need to "drive" the browser instead, you probably want to look into something like pyppeteer which drives a headless chrome/chromium
@rhymes
this is what I've tried to do:
I am still getting errors, any advice is welcome