Run Your Tests on the Cloud With TestCafe and BrowserStack

One of the best features the TestCafe testing framework has is its excellent browser support. When you write your tests using TestCafe, you can run them on most modern browsers right from the start without any extra configuration.

Having the support to handle different browsers out of the box helps get you started quickly. You can write a couple of tests, and fire up your test suite using any officially supported browser installed in your test environment.

However, chances are you will need to run your end-to-end tests on browsers that aren't readily available in your test environment. You might have the ability to install other browsers locally, but something this isn't feasible. For instance, if you only have access to Mac OS X or Linux operating systems, you can't easily set up Internet Explorer 11 (which is still actively used, much to the chagrin of web developers worldwide).

There's also the possibility that you need to run your tests in different environments altogether. While it's less frequent these days, some web applications only work on very specific versions of a browser available for a particular operating system. Also, there are rare cases where an app works differently in the same browser but under a different OS.

If you're serious about testing your web application, you need to work towards getting your app up and running under different circumstances.

Virtualization helps but still has its limits

These days, we have easy access to virtualization. For those unfamiliar with virtualization, it's the process of running virtual versions of devices and operating systems without the need of real hardware. All interactions and operations tied to hardware like storage and networking are abstracted away.

The use of virtualization in testing is on the rise. Using software like Oracle VirtualBox or VMware Fusion, you can set up a number of operating systems and browsers quickly on a virtual machine. Companies like Microsoft offer free virtual machines to help you test your web application in different versions of Windows and Internet Explorer / Edge.

Getting virtual environments up and running is simple, but it also has its limitations. Managing virtualization tools is tedious, and keeping them up to date is time-consuming. Between operating system and browser updates, both which receive frequent updates these days, staying on top of changes takes lots of time.

In addition, while virtual machines are helpful, they're not a substitute for real devices. There are tools to simulate the most popular operating systems for devices like phones and tablets. However, these tools only simulate the software, not the actual hardware. There are scenarios where you need a real device to replicate real-world hardware conditions.

Virtualization is a viable option to set up multiple environments for testing purposes. You might even supplement your virtual machines with real hardware devices. But if you don't have the time or resources to manage a virtual lab, what can you do?

Harnessing the Cloud for Web Application Testing

Thankfully, these days we have access to testing services that let us spin up any environment we need for our tests. One of the most popular testing services is BrowserStack, which is the focus of this article.

BrowserStack is a testing platform that allows developers to test websites or mobile applications across a variety of environments. They provide support for both virtual environments and access to real hardware. You can access both Windows and Mac environments, spanning just about every major OS and browser version for the past 10+ years.

To use BrowserStack, all you need to do is to generate an instance of an operating system and a browser or device, and you'll have access to it on demand. From here, you can do exploratory testing or run your automated tests.

Using BrowserStack with your TestCafe test suite

If you have your end-to-end tests working with TestCafe, it's simple to run your test suite on BrowserStack with little additional configuration.

The team behind TestCafe released an official plugin to run your test suite directly on BrowserStack. All you need to do is install the plugin in your project and set up a few environment variables.

First up, you need to have an active BrowserStack account with access to the Automate Pro tier. Keep in mind that BrowserStack isn't free, but they generously offer free access to open source projects.

When you have access to a BrowserStack account, you will need your Automate account username and access key. TestCafe needs this information to run your tests on the cloud service. Your account username and access key are available in your BrowserStack account settings page, under the Automate section. With this information, you can begin setting up the required plugin to run your TestCafe tests.

If you need an introduction to TestCafe, Dev Tester covered how to get started with TestCafe. For this article, we'll use those same examples to run the tests on BrowserStack. You can read the article to learn how to create the tests from scratch, or you can find the finalized code for that article on GitHub.

Using the code from the repository mentioned above, we can use it to install the plugin for our project. Inside the project's directory, install the plugin by running the following command:

npm install testcafe-browser-provider-browserstack

Note that we're installing the plugin in the current directory. The official documentation mentions to install TestCafe globally, but for simplicity, we'll keep it installed locally.

After installing the plugin, you'll need to set up a few environment variables to configure the plugin for correct use. The plugin uses different environment variables to control how you want to run your tests, but two variables are required: BROWSERSTACK_USERNAME and BROWSERSTACK_ACCESS_KEY.

As their names suggested, it's where you configure your BrowserStack username and access key. For Mac and Linux environments, you can set up these variables with the following commands:

export BROWSERSTACK_USERNAME="<BrowserStack Authomate username>"
export BROWSERSTACK_ACCESS_KEY="<BrowserStack Automate access key>"

(For more information on setting up environment variables on other environments like Windows, read this Twilio article or leave a comment below to receive help.)

With the plugin installed and your BrowserStack credentials set up, you can now begin to run your TestCafe test suite on BrowserStack. It's that simple! To verify that everything is working correctly, run the following command to fetch a list of available browsers and operating systems available for your tests:

npx testcafe -b browserstack

If you have everything set up correctly, you'll receive a huge list of available environments - over 2000, depending on your BrowserStack account.

The list indicates an available operating system and browser version. For example, "browserstack:Chrome@79.0:Windows 10" is an instance of Microsoft Windows 10 running version 79 of the Google Chrome browser.

If your BrowserStack account allows for testing using real devices, you'll also see them on this list. For instance, "browserstack:iPhone 11 Pro@13.2" is an Apple iPhone 11 Pro device running iOS 13.2.

If you receive an error while running this command, most likely your BrowserStack username or access key is not set up correctly. Verify that your BrowserStack credentials are valid, and you exported the BROWSERSTACK_USERNAME and BROWSERSTACK_ACCESS_KEY environment variables correctly.

Once you've confirmed everything is correct, you can begin to run your tests on BrowserStack with any the environments returned by the previous command. For instance, to run all of our tests in the test repository mentioned earlier in an instance of Mac OS X Catalina using Firefox 71, execute the following command:

npx testcafe "browserstack:firefox@71.0:OS X Catalina" "*_test.js"

TestCafe instantiates the specified environment on BrowserStack and runs all your tests automatically for you. You won't see the tests run on your local development environment, but you can see the results of the tests come in.

That's all you need to have instant access to thousands of environments to test your web apps.

Tips and tricks

Using the default settings, as shown above, is already useful by itself. Still, you can make this setup fit your needs even further.

Run your tests in multiple environments simultaneously

Another cool built-in feature in TestCafe is the ability to run tests concurrently, including using multiple browsers. This functionality is also available using the BrowserStack plugin, allowing you to run your tests in different environments at the same time.

To run your tests in different environments, add the desired environments as a comma-separated list in your TestCafe command. For instance, the following command runs your TestCafe test suite in Microsoft Edge 18 on Windows 10, Safari 13 on Mac OS X Catalina, and Google Chrome 79 on Window 7:

npx testcafe "browserstack:edge@18.0:Windows 10","browserstack:safari@13.0:OS X Catalina","browserstack:chrome@79.0:Windows 7" "*_test.js

The initial setup to spin up all the environments can take some extra time, but once the environments spin up, the tests run in each environment at the same time. It saves time on longer test runs by concurrently executing your tests.

Keep in mind that your BrowserStack account has a limit on the number of test instances you can run in parallel. If you exceed your limit, TestCafe won't run any tests. A workaround is to trigger your tests sequentially.

Make sure you run your tests in supported environments

With access to thousands of environments, it's tempting to want to automate your tests to execute in all sorts of conditions. However, there are a few drawbacks to this.

As a general rule of thumb, you should determine ahead of time which browsers and operating systems you want to target and limit your test runs to those. It keeps your builds speedy and makes it easier to manage.

There's rarely a need to test your web app using the same browser on different operating systems, or using incremental updates. Figure out what your audience uses and target their environments. It gives you the most "bang for your buck."

Additionally, there are a few environments that don't play well with TestCafe. There are some cases where TestCafe attempts to run your tests but eventually times out. It seems to happen with older browsers and operating systems, so if possible, try to stick to newer versions if you're having trouble with some environments.

Check out additional BrowserStack settings

As mentioned, you can do plenty with TestCafe's BrowserStack plugin by default. But BrowserStack has more settings that you can control via environment variables when running your tests.

Some of the more useful environment variables you can use are:

• BROWSERSTACK_USE_AUTOMATE: BrowserStack has two different APIs used for browser testing: JS Testing and Automate. Each API has its tradeoffs - the main ones being that the JS Testing API supports more devices while the Automate API has additional features like longer test runs and window resizing. By default, the plugin runs tests using the JS Testing API, but you can run your tests with the Automate API using by setting this environment variable (ROWSERSTACK_USE_AUTOMATE="1").
• BROWSERSTACK_TEST_RUN_NAME: This setting allows you to name your build, which is useful for keeping track of test runs on the BrowserStack dashboard.
• BROWSERSTACK_DISPLAY_RESOLUTION: You can set the display resolution for your BrowserStack environment with this variable. For example BROWSERSTACK_DISPLAY_RESOLUTION="1600x1200" sets the environments resolution to 1600 pixels by 1200 pixels. Note that this is the environment's display resolution, not the browser's resolution - for that, you can use TestCafe's window resizing API.
• BROWSERSTACK_GEO_LOCATION: This option sets the instance's IP to a specific geolocation, allowing you to test your application as if it were running in that location. For example, BROWSERSTACK_GEO_LOCATION="IN"simulates someone accessing the site in India. This option can be useful if your application behaves differently according to the user's location.

Some of these options are extra features on BrowserStack. Verify your account to see if you can use them.

Access your BrowserStack dashboard for debugging

When you run your tests on BrowserStack, TestCafe prints out the URL of the build returned by BrowserStack. This URL takes you to your BrowserStack account, where you can see the status of the build.

Here, you'll have a lot more information about the test run. For example, by default, BrowserStack records video of your test run, so you can see exactly how the test ran in your selected environment.

BrowserStack has other features you can set with environment variables to aid with debugging. For example, you can tell BrowserStack to capture any browser console logs or network information during your test runs. BrowserStack has many of these features disabled by default, so check what you need before running your tests.

Summary

TestCafe has excellent built-in support for all major browsers, with no extra setup or tweaking. As long as you have these browsers available in your test environment, your tests are ready to run.

It's not always feasible or possible to have all your necessary test environments ready to run locally. You can always use virtualization tools or even buy dedicated hardware to set up a lab of all the different environments your project needs. However, it's difficult and expensive to keep up to date.

Using cloud services like BrowserStack allows you to run your tests on thousands of environments on-demand. You don't need to spend time or money maintaining a lab of testing software and hardware.

Using TestCafe's plugin unleashes the power of BrowserStack for your test suite with minimal additional effort. Combining the usefulness of both TestCafe and BrowserStack covers all of your testing needs.

Do you have to test your web application in different environments? How do you handle running your application in a variety of scenarios? Leave a comment below and share your experiences!

P.S. Was this article helpful to you? I'm currently writing a book that will cover much more about about the TestCafe testing framework.

With the End-to-End Testing with TestCafe book, you will learn how to use TestCafe to write robust end-to-end tests on a real web app and improve the quality of your code, boost your confidence in your work, and deliver faster with less bugs.

For more information, go to https://testingwithtestcafe.com. Sign up and stay up to date!