I saw somewhere that Twitter can load with JavaScript disabled "only for some User Agents".
Indeed, making your website require JavaScript does kill many high performance web scraping.
Thank you for the link @Pacharapol, I'll definitely read the article.
Do you have any resource that I can check for this? Or is it in the linked article?
No, I cite it myself.
Disable JavaScript means you cannot use merely GET web scraping (e.g. Python requests), and read the HTML output (e.g. Python BeautifulSoup). You will need to resort to more heavyweight method like Selenium or Puppetteer (Chrome engine).
GET
Excellent. Thanks for the info.
Some comments have been hidden by the post's author - find out more
Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink.
Hide child comments as well
Confirm
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
I saw somewhere that Twitter can load with JavaScript disabled "only for some User Agents".
The importance of use User Agent to Scraping Data
Hugo Sandoval ・ Apr 30 ・ 6 min read
Indeed, making your website require JavaScript does kill many high performance web scraping.
Thank you for the link @Pacharapol, I'll definitely read the article.
Do you have any resource that I can check for this? Or is it in the linked article?
No, I cite it myself.
Disable JavaScript means you cannot use merely
GET
web scraping (e.g. Python requests), and read the HTML output (e.g. Python BeautifulSoup). You will need to resort to more heavyweight method like Selenium or Puppetteer (Chrome engine).Excellent. Thanks for the info.