Continuous Testing with Travis CI and LambdaTest

When you build any software program, one of your goals is to have as many people using it without any issues. If your app has too many problems that impede the vital functionality you want for your purposes, no one's going to stick around. That's why testing and test automation are essential parts of a product's lifecycle.

With web applications, testing can get complicated quickly. Often, it starts with a few manual tests in a specific environment. It allows your team to get a head-start when you decide to move to automated tests. Once you have some automation going on, your testing efforts expand to cover the most-used platforms. Test automation allows you to validate your functionality across different environments with ease.

However, even if you cover the main browsers and devices in use these days, this amount of testing isn't enough. Once your app is out there in the real world, you'll find a seemingly infinite number of combinations of platforms. You not only have to manage different versions of the web browsers you test your app against, but you'll also have to deal with them in varying combinations of devices and operating systems.

Most organizations use virtualization to spin up the environments that they need for testing purposes. These days it's dead-simple to run virtualized systems. However, you'll need to maintain these systems, and it's often a pain to manage when you scale up. Also, virtualization is not a substitute for real hardware. You might run into scenarios where you'll need the real thing to replicate a problem and fix it.

Instead of wasting time and money handling different environments to test your web application, you can use LambdaTest to address your cross-browser test automation needs.

LambdaTest - The most powerful cross-browser testing tool online

LambdaTest is a cloud-based testing platform that allows teams to test websites or mobile applications in different environments, using both desktop browsers and browsers on mobile devices. With just a few clicks and some minor configuration, you can spin up any operating system and browser combination to have a usable system to run your tests.

You can access any operating system from Windows XP to Windows 10 and all Mac OS X editions since 2011. You can also use any version of most-used web browsers from the past 15 years. Whether you need to ensure that your application runs in older environments or on new systems, LambdaTest can handle it for you.

Here are a few of the things you can do with LambdaTest to boost your test automation efforts:

• Perform manual and automated tests on your web application, using both virtualized environments and real hardware devices.
• Automatically generate screenshots of your application across multiple systems to perform visual regression testing.
• Validate the responsiveness of your web app across multiple iOS and Android devices.
• Run your tests on a scalable Selenium Grid system with access to over 2000 desktop and mobile browsers and operating systems.
• Test from more than 27 countries to ensure everyone across the globe can reach your application.
• Integrate with many third-party applications like Jira, TestRail, and Slack to keep your testing efforts in sync across your organization.

LambdaTest provides lots of other excellent tools to help with your testing organization. The LT Browser aids with responsive testing during development. There's an issue tracker to keep track of what needs fixing inside the service. If you use Google Chrome, a Chrome extension quickly sends screenshots of your app during manual testing.

I've used similar cloud-testing services in the past, but most of them only offer a fraction of the tools that LambdaTest provides. It's a complete package that covers almost all of your testing needs.

LambdaTest in action

In this article, I want to demonstrate how LambdaTest works by running a continuous testing workflow for an existing web application. For this demonstration, I'll use an automated end-to-end test suite built with the TestCafe testing framework. Although LambdaTest's infrastructure is built on Selenium Grid, we can also use other testing frameworks on the platform for test automation, even if they don't use Selenium under the hood.

The rest of the article assumes you have an active LambdaTest account set up. If you don't have an account and want to follow along, you can register for free. LambdaTest has a free tier with limited access that allows you to use all of their services. It's perfect for helping you get started.

Setting up TestCafe and LambdaTest

The test suite we'll use in this article comes from my book, End-to-End Testing with TestCafe. It contains 11 end-to-end tests covering different functionality for the book's companion web application, TeamYap.

Setting up an existing TestCafe test suite to use LambdaTest's automation services is straightforward. The LambdaTest team maintains a TestCafe plugin that allows you to run your TestCafe tests on the LambdaTest platform. To install the plugin, run the following command inside your test project directory:

npm install testcafe-browser-provider-lambdatest

All the configuration for the plugin is handled by setting up environment variables. Using environment variables allows you the flexibility to set up your automated test runs differently, depending on the environment, without the need to modify any files. Most of LambdaTest's settings have sensible defaults, and you probably won't need to change them. You do need to set up two environment variables that are required for the plugin to work:

• LT_USERNAME - Your LambdaTest username, automatically generated for you after signing up.
• LT_ACCESS_KEY - Your LambdaTest account's secret access key.

Your LambdaTest username and access are available in your account profile settings. Once you have this information, you can set the required environment variables with the following commands in your preferred terminal on Mac OS X and Linux:

export LT_USERNAME=<LambdaTest Username>
export LT_ACCESS_KEY=<LambdaTest Password>

For Windows systems, use the following commands through the Command Prompt or PowerShell:

set LT_USERNAME=<LambdaTest Username>
set LT_ACCESS_KEY=<LambdaTest Password>

To verify that your LambdaTest credentials are set up correctly, you can check the list of available browsers and operating systems provided by the LambdaTest service through TestCafe using the following command:

testcafe --list-browsers lambdatest

If your credentials are correct, this command prints a long list of browser aliases that you can use to run your TestCafe tests on LambdaTest. The browser aliases are namespaced with lambdatest, and contain the combination of the browser and operating system for both desktop and mobile devices. For instance, the browser alias "lambdatest:Firefox@78.0:Windows 10" is for a Windows 10 system using Firefox 78, and "lambdatest:Safari@13.0:MacOS Catalina" is for a Mac OS X Catalina (10.15) system using Safari 13. If you don't see a list of browser aliases, double-check your credentials and set them appropriately.

Running tests on LambdaTest

Now that you have LambdaTest and TestCafe set up, all that's left to do is execute your tests on the service. Thanks to TestCafe's plugin system, the command to perform this action is identical to running your tests locally:

testcafe <LambdaTest browser alias> <test files>

For example, if you want to run all of the tests on the project on Mac OS X Catalina using an instance of Microsoft Edge version 84, you can use the following command:

testcafe "lambdatest:MicrosoftEdge@84.0:MacOS Catalina" *_test.js

This command automatically sets up a tunnel between your system and LambdaTest, and triggers your test using the specified environment on the cloud. TestCafe receives the results for each test as they complete. The reporter plugin also shows a link to the LambdaTest session so that you can keep track of the test run on the service.

While the tests run, you can check the LambdaTest dashboard to see your test runs and current sessions' stats.

When the test run finishes, LambdaTest closes the session and keeps a record of your test run. In the Automation section of your LambdaTest account, you'll see a timeline of your builds and automated test runs. Here, you can access all of the test run information, including a video recording of the test run on LambdaTest's system.

When you use the TestCafe plugin to run your tests without modifying any additional settings, it will use the following:

• The build name used is "Untitled".
• The test run name used is automatically generated by TestCafe.
• The screen resolution is 1024x768 pixels.

As mentioned earlier, you can modify LambdaTest's default settings through environment variables:

• LT_BUILD - The name of the build to use for this automated test run.
• LT_TEST_NAME - The name you want to use to identify your test run.
• LT_RESOLUTION - The screen resolution for the LambdaTest environment, defined by width and height in pixels (like "1600x1200", for instance).

You can also modify other settings, including logging JavaScript console messages, record network packets, and disable video recording through other environment variables. See LambdaTest's TestCafe plugin documentation for more details, and LambdaTest's Selenium documentation for information on the platform's capabilities.

Continuous testing with LambdaTest and Travis CI

It's pretty nice having your TestCafe tests running on the cloud with LambdaTest. The LambdaTest plugin makes the entire process effortless. However, we can take it a step further by setting up continuous integration to automatically run your tests. If your project receives an update, your CI service can run your tests and alert you immediately if the changes cause any regressions.

To show how continuous testing works, we'll use one of the most popular continuous integration services: Travis CI. Travis CI is a hosted continuous integration service that syncs with your code repository to get your test automation up and running in no time. This article will show you how to connect your TestCafe code repository to TravisCI and set it up to run your tests on LambdaTest automatically.

TravisCI is free for open-source projects, and signing up is as simple as logging in with your preferred source code management provider. TravisCI supports GitHub, GitLab, Bitbucket, and Assembla, so if your project's code is on one of these platforms, you can automatically connect it to TravisCI.

After signing up with your source code management provider, TravisCI syncs up with your account to fetch your available code repositories. You can visit your account settings to search for the repo you want to connect with TravisCI and click on the toggle switch next to the repo name. Turning the toggle switch to "on" sets up your code repo with a webhook that notifies TravisCI of any changes.

Before you can use TravisCI to build your project, you need a bit of configuration. First, you need to set up the environment variables that LambdaTest needs, as discussed earlier in this article. TravisCI makes it easy to set up environment variables securely.

When you turn on the project inside TravisCI, click on the "Settings" button next to the toggle switch. You can set up the environment variables your project needs to run the build properly among the project settings. Here, you can set up the LambdaTest environment variables. Also, the TestCafe test suite uses a few environment variables for the account credentials throughout the test scenarios, so we need to set them here.

The final step in the process is to configure your project to tell TravisCI what it needs to do for every build. You can manage this configuration through a build configuration file called .travis.yml file, placed inside your code repository. In this file, you need to specify the language support for your project, at a minimum. Depending on the language, TravisCI will run a few commands by default, but you have control over the entire process if needed.

For our project, TestCafe is a testing tool that uses Node.js under the hood, so we can set our build configuration to use a Node.js platform when running builds on TravisCI. It's also a good practice to specify at least one Node.js version to ensure that your tests run on a known working version. Inside of the TestCafe test suite project, you can create the .travis.yml file in the root of the project with this configuration:

language: node_js
node_js:
  - 12

By defining node_js as the language for this project, TravisCI automatically runs the following commands:

• npm install to install all of the project's dependencies.
• npm test to run your tests automatically.

The project already has a valid package.json file containing the dependencies needed to run the TestCafe test suite on LambdaTest. The file also has the test script with the same command used earlier, so no further configuration is necessary with TravisCI. Here's the completed package.json file for reference:

{
  "name": "end_to_end_testing_with_testcafe",
  "version": "1.0.0",
  "description": "Demo repo showing how to run TestCafe tests using LambdaTest and TravisCI",
  "main": "login_test.js",
  "scripts": {
    "test": "testcafe 'lambdatest:MicrosoftEdge@84.0:MacOS Catalina' *_test.js"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/dennmart/lambdatest_travisci_testcafe.git"
  },
  "author": "Dennis Martinez",
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/dennmart/lambdatest_travisci_testcafe/issues"
  },
  "homepage": "https://github.com/dennmart/lambdatest_travisci_testcafe#readme",
  "dependencies": {
    "testcafe": "^1.8.7",
    "testcafe-browser-provider-lambdatest": "^2.0.2"
  }
}

With the build configuration file in the project and TravisCI connected to the code repository, your tests will now run after every repo change. To test this out, push the .travis.yml file to the remote repository. If everything is set up correctly, TravisCI will automatically trigger a build after a couple of seconds.

The build starts by setting up the environment variables, installing all project dependencies, and runs the test script. With the environment variables in place, your tests will run as they did before. The LambdaTest plugin creates a tunnel to its service and runs the test in the specified operating system and browser.

With some minor setup, you can leverage the power of LambdaTest to execute your TestCafe tests in any environment, using TravisCI to run them for you automatically.

Summary

Testing your web application on a combination of operating systems, web browsers, and devices can be a hassle. It's costly and time-consuming to maintain different environments to ensure that your app works. With LambdaTest, it doesn't have to be a pain.

LambdaTest provides an easy yet powerful way to test any web application in a variety of setups. It provides all the tools to ensure your app works as intended, from spinning up different operating systems and browsers on-demand to performing other testing duties like responsive and visual testing.

This article shows how simple it is to set up a continuous testing workflow with just about any modern environment, thanks to LambdaTest and TravisCI. The examples shown here are for running a TestCafe suite, and LambdaTest provides a dead-simple way to run your tests on the cloud. If you don't use TestCafe, LambdaTest also supports lots of other languages and frameworks, so chances are you'll have support for your existing test suite.

These days, you have plenty of tools at your disposal to make your test automation journey easier. LambdaTest is one of the most robust and powerful services available to help you tackle any potential issues before your users encounter them. It'll improve the quality of your application with little extra effort.

Have you used LambdaTest with your projects or in your organization? Share your experiences in the comments below!

This post is sponsored by LambdaTest, a cloud-based cross-browser testing tool to perform manual or automated testing on over 2000 browsers online. All reviews and opinions expressed in this article are my own and based on my personal experience with the service.