Web Automation with Python and Selenium

Web automation is one of the best ways companies can test a product in development, especially the app's functionalities, such as clicking, scrolling, and other actions. 

Essentially, web automation is about mimicking human action as it is crucial to ensure the software works across all device types of users.

In this article, we will learn how to use Selenium as an automation tool to test a website with Python and automate the entire process without using a mouse or keyboard on the browser. Selenium doesn’t work with Python alone but with many other programming languages. 

What is Selenium Used For?

Selenium allows us to browse or use a browser without a human involved and automate processes through code, such as typing into a user input and interacting with the website.  For example, automating form submissions with Selenium is possible. Selenium does everything all by itself without a single click from a human.

Getting Started

Before Selenium can carry out any action through code, we need the packages and tools to be able to run browser automation and make it seamless. 

Installation

This section, we install the Selenium package and the WebDriver executable file. First, check the version of the Chrome browser so that it is equivalent to the same version of the WebDriver exec file.

The following needs to be installed on our local machine:

The Chrome browser. Download it here.

Download the latest stable release of the ChromeDriver. On the ChromeDriver web page, check to confirm the version of Chrome matches the version of the ChromeDriver.

Now, run this command to install Selenium in your terminal:

pip install selenium

Automating a Web Page

To begin, create a new folder containing the file and the downloaded ChromeDriver exec files. Selenium requires the driver to interface with the chosen browser, which in this tutorial, we are using the Chrome browser.

It is essential to place the ChromeDriver in the same directory as the Python file.

The directory for the project should look like this:

.
├── automation.py
└── chromedriver

Another way to use the ChromeDriver in this project is to find the path location of the file and copy it, which should look like this:

MacBook users:

/Users/<computer user name>/Desktop/chromedriver

Windows users:

C:\Users\<computer user name\Desktop\chromdriver

The WebDriver is the bridge that bridges the Selenium code to work with the Chrome browser. The Chrome browser provides the bridge. Without it, automation will not be possible.

Next, let’s copy and paste the following code into the automation.py file:

automation.py

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

chrome_driver_path = './chromedriver'

service = Service(chrome_drive_path)
driver = webdriver.Chrome(service=service)

url = "https://en.wikipedia.org/wiki/Main_Page"
driver.get(url)

article_count = driver.find_element(By.CSS_SELECTOR, "#articlecount a")
print(article_count.text)

drive.close()

The following occurs in the code snippet above as we will use Selenium to navigate Wikipedia:

• Import the function webdriver from the selenium module
• The selenium.webdriver module provides all the WebDriver implementations, and the Service object is for handling the browser driver
• The By class is used to locate elements within the document of a web page using its CSS classes, ids, and so on
• The Keys class simulates the actions from the keypress of the keyboard like the RETURN / Enter, F1, Alt, and so on
• Assign the chromedriver exec file to a variable
• Call the executable path of the driver to the service object
• Next is to instantiate the Chrome browser with the webdriver.Chrome() with the service argument
• Define the url of the web page to check
• Using the .get() method will navigate to the page given by the url

On the Wikipedia page, open the inspect element to access the elements we will use to get the details from the page for automation.

Now to the task of finding the article count. WebDriver offers us a way to do so by targeting the element using the find_element method and using the By.CSS_SELECTOR as the first parameter with the selector name, #articlecount a.

Before we run the Python program, the script has the method, driver.close(), that closes the browser window after running the test.

Run the program with the command:

python automation.py

The result from the terminal should display the article count, which shows the exact figure on the web page, 6,545,457.

PS: This value of the article count won't remain the same as its value would have increased.

Simulating a Click Action

From what we have done so far, it is evident that Selenium navigates a website without the use or click of a mouse.

Now, let’s simulate an action where the browser reacts to a click of a link on the page.

Update the automation.py with these lines of code:

automation.py

...

contact_us = driver.find_element(By.LINK_TEXT, "Contact us")

contact_us.click()

With this code snippet, we are finding the element to the link text, “Contact us” with the By.LINK_TEXT. After locating the elements, attach the contact_us to an action with .click() method.

Again, run the command:

python automation.py

The command above gives the result of the contact us page.

Search a Web Page

There are so many use cases with Selenium. Searching a web page can be achieved with automation, which gives you the result for a search word.

Still, in the same file, automation.py, update the file with this code:

automation.py

search = driver.find_element(By.NAME, "search")

search.send_keys("React", Keys.ENTER)

The code above does a similar thing to the previous by using the find_element method with the input's name attribute, search.

Also, the send_keys simulate pressing the ENTER key, searching the keyword React.

The result of the page should look like this:

Conclusion

This article is an overview of how to use Selenium in Python to automate the web, especially for professional developers who want to know if there are errors in the code and how to fix them.

The possibilities of using Selenium with Python for web automation are endless. Some use cases include filling out an online form, applying for LinkedIn jobs, playing a clicker game, and so on. Do you know other use cases for using Selenium? 

Kindly share your favorite use case in the comments below.

Learn More

Selenium documentation
XPath Tutorial