Automating Data Extraction with Selenium & SQLite

Objective: Present a simple use case of data extraction with Selenium and storing in a SQLite database, where files are downloaded and can be analyzed for new insights.

Code

I created a Python script that accesses the StatCounter website and downloads all statistical data on platform usage in Brazil over the years.

The script is available here. Make sure you have the necessary libraries installed - read the README.md - and Selenium configured with Chrome Driver.

How does it work?

1. Chrome Headless Configuration: Selenium is configured with Chrome's headless option to automate website navigation without a graphical interface. This is handled by the configure_chrome_headless() function.

2. CSV Download and Parsing: Using the requests library, the script downloads each CSV file containing the statistical data. Then, the csv library is used to parse and extract relevant information. This is represented by the download_and_parse_csv() function.

3. Storage in SQLite: Tables are created and updated in an SQLite database, organizing the data by year. This is handled by the save_to_sqlite() function.
   

python platform_market_share.py

I hope to bring more use cases exemplifying the use of Selenium with Python code, as there are many other interesting things to do with these technologies. Share by commenting below any example of a case you know or have developed!