Comparison of Automation Frameworks: Selenium, Playwright and Cypress

In modern software development, test automation has become an indispensable element, enabling QA teams and developers to efficiently and effectively verify the quality of their applications. Several tools have emerged to facilitate this process, such as Playwright, Selenium, and Cypress being some prominent options. This comparison will analyze and contrast the three leading tools in test automation, showcasing their strengths, weaknesses, and distinctive features.

Selenium: Widely used, it supports multiple languages and browsers, ideal for end-to-end testing.

Cypress: Modern approach, designed for in-browser testing, stands out for its simplicity, speed and efficiency in integration and unit testing.

Playwright: It is versatile, allowing the automation of advanced tasks, such as emulation of mobile devices. It stands out for its ability to work in multiple browsers and run test in parallel.

General information

In the following table you will learn the general information for each framework:

Selenium is the oldest and still widely used framework/library for Automation. Playwright, on the other hand, is the newest framework, but their popularity is increasing quickly in the community. Microsoft created Playwright to address common challenges in Selenium, as well as improve integration with DevOps tools and cloud services.

The main feature in Playwright is the performance and support for modern programming languages and frameworks. Selenium has the best support for old browsers and programming languages. Cypress has only support for the JavaScript ecosystem.

Popularity

Let's review the popularity and numbers of downloads in npm. Check the following picture:

In the last months, Playwright has overcome selenium in the number of downloads, stars and contributors on GitHub. This means a great appreciation by the community to this framework.

Selenium is widely regarded as the top choice for Java and Python due to its extensive implementation in major projects over many years.

Demos

In the following sections, we will reproduce the first steps to create our first automation test for these 3 technologies.

For these demos we will use the following web page: https://www.saucedemo.com/

The idea is to perform the login and validate that it was successful on the following screen:

Cypress

Let's review the prerequisites and commands to create our first test in Cypress.

Prerequirements

• Node.js 18 or 20

Steps:

• Go to an empty folder where you want to create the project
• Execute npm install cypress --save-dev
• Then use this command to execute the project: npx cypress open
• Now in the folder cypress/e2e create a new test with the name spec.cy.js
• Add the following code to create a new test for the login.

describe('Sauce demo', () => {
  it('login pass', () => {
    cy.visit('https://www.saucedemo.com/#/')
    cy.get('#user-name').type('standard_user')
    cy.get('#password').type('secret_sauce')
    cy.get('#login-button').click()
    cy.get('.app_logo').should('have.text', 'Swag Labs')
  })
})

In cypress, we type cy to use the API and call the functions to detect elements, set values and navigate in the page. The code is petty simple and easy to understand. We use #element-id cal get elements by Id and .class to get the element by CSS class. We don't need to install extra libraries or components to create our first tests.

Selenium

Let's review the prerequisites and commands to create our first test in Selenium with JavaScript.

Prerequirements

• Node.js 18 or 20

Steps:

• Go to an empty folder where you want to create the project

• Execute the following commands:

  npm install selenium-webdriver

  npm install chromedriver

  npm install mocha

NOTE: selenium-webdriver is used to interact with the browser, chromedriver is used to execute the test in chrome so Selenium can control it and mocha help us to create the test like a simple unit test and capture the result.

• Create a file test.js where the test will be located
• Add the following code to perform the test on the login page with Selenium:

const Chrome = require('selenium-webdriver/chrome');
const {By, Builder} = require('selenium-webdriver');
const assert = require("assert");
const options = new Chrome.Options();
const {suite} = require('selenium-webdriver/testing');


suite(function (env) {
    describe('Should be able to Test Command line arguments', function () {
      it('headless', async function () {
    let driver;

    driver = await new Builder()
            .setChromeOptions()
            .forBrowser('chrome').build();
    await driver.get('https://www.saucedemo.com/');
    await driver.manage().setTimeouts({implicit: 500});
    await driver.findElement(By.id('user-name')).sendKeys('standard_user');
    await driver.findElement(By.id('password')).sendKeys('secret_sauce');
    await driver.findElement(By.id('login-button')).click();
    var textLogo = await driver.findElement(By.className('app_logo')).getText();
    assert.equal('Swag Labs', textLogo);
    await driver.quit();
});
});
});

Selenium has a better experience in other programming languages, like Java, Python and C#. Since Selenium is a library, we need to install other libraries to complement it. And install the drivers to connect with the browser where we want to run the test.

Playwright

Let's review the prerequisites and commands needed to create our first test in Playwright with JavaScript.

Prerequirements

• Node.js 18 or 20

Steps:

• Go to an empty folder where you want to create the project
• Run npm init playwright@latest to create the base project
• Then use this command to execute the tests: npx playwright test
• Create a new file mytest.spec.js in the folder tests
• Add the following code to test the login page using Playwright:

const { test, expect } = require('@playwright/test');

test('Login pass', async ({ page }) => {
  await page.goto('https://www.saucedemo.com/');
  await page.locator("#user-name").fill("standard_user");
  await page.locator("#password").fill("secret_sauce");
  await page.locator('#login-button').click();

  // Expect a title "to contain" a substring.
  await expect(await page.locator('.app_logo')).toHaveText('Swag Labs')
});

In playwright, we use the word page to interact with the browser. We can also use the method .toHaveText to validate the result and create assertions. Playwright is not only incredibly user-friendly for developers, but it also boasts a seamless functionality that requires no additional dependencies or extra libraries in order to work perfectly in a variety of scenarios.

Developer Experience

JavaScript's developers find Cypress easy to understand because the tests are similar to unit tests in JavaScript with libraries like jest. Cypress may be limiting and restrictive for automation QA professionals who prefer to have full control over their workflow and do not rely on the UI for test inspection and execution. Cypress only works with JavaScript, so it's hard to compare it with Playwright, which is popular among other developers.
Selenium has a high flexibility and I can combine it with my favorite libraries. Once the project is established, we can enhance the test structure, streamline the main framework, and optimize code reuse. This approach is favored by many developers.

UltimateQA posted a survey about the user experience for these frameworks. Ultimate QA survey

The results seem inaccurate because Cypress and Playwright are incredibly user-friendly, yet Selenium remains the most widely used automation library and many developers only have familiarity with it.

Possible Scenarios vs Best Option

Let's analyze some common scenarios and which framework is a better fit.

1. A modern application with a small team that uses a new Javascript framework (React.js, Angular, etc..). The team doesn't have experience with automation testing. For this scenario, the best option could be Cypress or Playwright. Developers can easily work with the UI provided for these frameworks and work with JavaScript.

2. An app with a modern backend (Node.js, Python, C#, etc.) and modern frontend (React.js, Angular, etc.) with a big team.
   For this scenario, we can use all the 3 options. But to work with something modern Playwright could be the best one. We can work with different languages according to the team knowledge and Playwright has been adding extensibility with Selenium.

3. A desktop application using Electron (JavaScript) with node.js backend. In this case, all these frameworks are bad options. At the moment, both Playwright and Cypress offer experimental plugins or extensions for electron, but they may not be suitable for production purposes. We can use Spectron, WebDriverIO or Selenium in conjunction with other libraries to perform our tests.

Conclusions

In modern software development, the main test automation tools are Selenium, Cypress, and Playwright. Each of these tools offers unique strengths and weaknesses, catering to different needs within the software testing landscape.

Selenium, as the oldest and widely used framework, excels in supporting multiple languages and browsers, making it ideal for end-to-end testing. Cypress, with its modern approach, stands out for in-browser testing, simplicity, and efficiency in integration and unit testing. Playwright is a versatile automation tool that excels in multiple browser support, parallel test execution, and automating advanced tasks like mobile device emulation.

The popularity metrics, including downloads, stars, and contributors on GitHub, depict Playwright's recent surge, surpassing Selenium in community appreciation. However, Selenium remains a top choice for many teams.

Ultimately, the choice of test automation tool depends on project requirements, team expertise, and the application being tested.

In my opinion, Playwright will continue growing, and it is a modern and fast tool. I see a great projection for Playwright in the future.

If you want to try the demos and access the repo, click on the following link:
https://github.com/Mteheran/AutomationDemos