Migrating Selenium system tests to Cuprite

In our project, we’ve been running system tests (then called rather "Feature tests") since around 2016. System tests use a real browser in the background and test all layers of a Rails application at once: from the database all the way up to the nuances of JavaScript loaded together with the web pages. Back then, we wrote our system tests using Capybara with Poltergeist, a driver that ran a headless Phantom JS browser. Since this browser stopped being actively developed, we migrated our test suite to the Selenium / Webdriver wrapper around Chrome browser around ~2018. Chrome was itself fine for tests automation but the Selenium API was quite limited and we had to rewrite several Poltergeist features using 3rd party gems and tools.

That is why we were happy to find out that a new ruby testing driver approach is being developed. It is called Cuprite, it runs the Ferrum library under the hood which, in turn, is an API that directly instruments the Chrome browser using the Chrome DevTools Protocol (CDP). About a week ago, we finally made a serious attempt to make our system test suite run on Cuprite, with especially two questions in our minds:

• would the tests run faster?
• would the Cuprite API be easier to use?

As a little spoiler we are glad to say that both points turned true for us and we kind of fell in love with these wonderful pieces of software, Cuprite and Ferrum. If you’d like to hear more details, read on.

The migration

All important parts of the basic installation process are shown in the Cuprite README and in the customization section of the Ferrum README. Great resources and tips can also be found in this article by Evil Martians.

The very lovely thing about Cuprite is that it very much resembles the old but good Poltergeist API. The CDP protocol is much more versatile than Selenium driver and thus Cuprite allows e.g. the following things which were hard or even impossible with Selenium:

• blocking / allowing requests to external domains and URLs,
• setting cookies, even before visiting the given page,
• setting request headers,
• opening the Chrome DevTools with a single line of code,
• inspecting and/or logging all communication between the test and the Chrome browser (all CDP messages).

For a lot of these features, we previously had to adopt various 3rd party gems, such as the Puffing Billy proxy (for blocking domains), the webdrivers gem (for auto-updating the Chrome drivers), etc. and although they certainly did a good job for us, now we were able to finally rip them off the project completely:

The Cuprite speed-up is real and can be helped even more

OK, let’s talk numbers. We have ~140 system tests in our project, covering the most important use cases in our web application. Several of the test cases go through some very complex scenarios, slowing down the whole test suite run time considerably. Overall, our system tests used to run approximately 12 minutes on Selenium, while the same suite finishes in ~7 minutes under Cuprite. That is approximately a 40% speed-up 😲!

Not all of this can be attributed to Cuprite speed alone though as, in the end, we configured the new driver slightly differently. For example we used whitelisting of specific domains instead of blocking the others as we did on Selenium. It is now a much stronger and stricter setup that probably blocks more domains than before, speeding up the page loads. Still, the speed up was clear and apparent since the first run of Cuprite.

Faster sign-in in tests

And we added a few more tricks. We rewrote our sign-in helper method in a more efficient way. This was possible because Cuprite allows setting a cookie (i.e. the session cookie) even prior to visiting a page, unlike Selenium. Thus, we could manually generate a session token and store it both to our back-end session store as well as the session cookie. We just needed to make sure the session cookie had the same options as the real session cookie.

def login_via_cookie_as(user)
  public_session_id = SecureRandom.hex(16)
  page.driver.set_cookie("session_test", 
                         public_session_id, 
                         domain: ".example.com", 
                         sameSite: :Lax, 
                         secure: true, 
                         httpOnly: true)

  private_session_id = Rack::Session::SessionId.new(public_session_id)
                                               .private_id
  Session.create!(session_id: private_session_id, 
                  data: { user_id: user.id })
end

This lead to another noticeable speed-up of the tests suite run.

Test fixes needed

Initially, about 30 tests (~20%) that were OK under Selenium, failed under Cuprite. Some of the failures were easy to fix, others were more puzzling. Overall, we came to a feeling that the Cuprite driver was less forgiving than Selenium, forcing us to be a bit more precise in our tests.

For example, we filled a value of "10 000" into a number input field in a test (note the whitespace). This works without issues inside Selenium but fails under Cuprite. Now, let’s show a few more types of fixes that we had to deal with.

Scrolling and clicking issues

A lot of tests failed because Cuprite tried to click an element that was covered by another element on the page. Cuprite seems to scroll and center the element a bit less (compared to Selenium) prior to clicking it.

Here is a typical example – the test was trying to click on the button covered by the sticky header, as we could easily see by saving the page screenshot upon test failure:

The failure log message would show a Capybara::Cuprite::MouseEventFailed error with details about which element was at the same position as the clicked-on element.

We had to manually scroll to an element in a few tests. To further mitigate this issue in a more generic way, we also overloaded the click_button method from Capybara to scroll and center the button on the page before clicking it:

def click_button(locator, *options)
  button = find_button(locator, *options)
  page.scroll_to(button, align: :center)
  button.click
end

File uploads needed absolute file paths

We use Dropzone JS to support uploading files. Under Cuprite, uploading stopped working and an ERR_ACCESS_DENIED error was shown in the JavaScript console each time a test attempted to upload a file.

It took a while to debug this but in the end the issue was quite prosaic – Chrome needed absolute paths when simulating the file upload in the test. So, the fix was just along the following lines:

- attach_file("file-input", 
-             "./app/assets/images/backgrounds/brown-wood-bg_512.png")
+ attach_file("file-input", 
+             Rails.root.join("app/assets/images/backgrounds/brown-wood-bg_512.png").to_s)

We are not sure if this issue is only when using Dropzone or rather related to generic file uploads in system tests.

AJAX / Fetch issues due to Cuprite being ”too fast“

Surprisingly, some more tests started failing randomly. Soon it turned out that all of them deal somehow with JavaScript sending requests to the back-end via AJAX or Fetch. Again, these tests were rather stable under Selenium and – as we investigated – the issue was that under some circumstances the Cuprite driver generated multiple Fetch requests and sent them too fast.

For example, we have a few ”live search“ fields, backed by back-end Fetch requests, on some pages. The live search function was usually triggered by the keyup event and Cuprite was such a fast typewriter that it frequently sent multiple requests almost at once. If some of the responses got a bit late or out of sync, the front-end JavaScript code began hitting issues. We solved this by adopting a technique called debouncing and, frankly, we should have done this since the beginning. By the way, we used the useDebounce module from the marvelous Stimulus-use library to achieve this.

Custom Cuprite logger

A lot of our migration effort went to developing a logger for some of the events that Cuprite / Ferrum handles when talking to the browser. In general, Cuprite offers a stream of all CDP messages exchanged between the driver and the browser. To use it, one has to filter out the events that he or she is interested in.

We used this feature to track two kinds of data in the log:

• JavaScript errors printed in the JS console in the Chrome browser,
• details about the requests and responses sent to/from the server as this information sometimes greatly helps debugging tests.

Usually, we let the test fail when a JavaScript error occurs. Ferrum has a js_errors option in the driver configuration to do just that. It works nice but we used a custom solution instead because we wanted some of the JavaScript errors to actually be ignored and we didn’t want a test failure then. In the end, we made a helper class (similar to this one) that collected all JS errors during a test run and checked this array of errors in the after block, allowing for ignoring preconfigured types of errors. Note that care must be taken about cleaning-up the state in RSpec as triggering an expectation error in the after block otherwise skips all later code in the block.

def catch_javascript_errors(log_records, ignored_js_errors)
  return if log_records.blank?

  aggregate_failures "javascript errors" do
    log_records.each do |error|
      next if ignored_js_error?(error, ignored_js_errors)
      expect(error).to be_nil, "Error caught in JS console:\n#{error}"
    end
  end
end

RSpec.configure do |config|
  # this is run after each test
  config.after do |example|
    catch_javascript_errors(page.driver.browser.logger.error_logs, 
                            example.metadata[:ignored_js_errors])
  ensure
    # truncate the collected JS errors
    page.driver.browser.logger.truncate
    # clean up networking
    page.driver.wait_for_network_idle
  end
end

Other CDP protocol events (namely Network.requestWillBeSent, Network.responseReceived and Network.requestServedFromCache) served as the basis for logging all requests and their responses. We chose a custom log format that enables us to better understand what’s going on – network wise – in each test and if you’re curious, it looks like this:

Summary

We are indeed very happy about the migration to Cuprite. Our tests are much faster, the API to handle them is simpler and the migration forced us to take a closer care while handling some special situations, benefiting not only the tests but the users visiting our site, too. Overall this feels like a great move and we heartily recommend anyone to do the same! 🤞

If you like reading stuff like this, you might want to follow us on Twitter.