DEV Community

Ben Halpern for The DEV Team

Posted on

Cypress vs.....

Hey folks, we're considering adopting Cypress for E2E testing on DEV, but want to get some feedback if anyone wants to propose other tools in this genre.

We currently do some E2E in rspec (Rails), but it's also a component of the app and process which is under-developed.

So feedback is much appreciated, thanks a lot!

Top comments (47)

Collapse
 
ccleary00 profile image
Corey Cleary

I'd take a look at TestCafe too. It's similar to Cypress in that it's a non-Selenium based E2E testing tool.

Anecdotally, at my job we initially tried using Cypress but found some limitations that were unfortunately deal-breakers. Cypress seems to have an opinionated view of how you should write your tests. After switching to TestCafe we haven't had any problems.

A downside to Cypress in my opinion is that it seems like it's trying to do too much. In addition to being a full E2E tool, you can write unit/integ tests with it. Which is really nice, but it's hard enough to get that piece of the puzzle right let alone get the E2E piece right too. Mocha and Jest, for example, are solely unit/integ testing tools and both of those tools still have issues...

Collapse
 
nickytonline profile image
Nick Taylor • Edited

It’s true you could write integration and unit tests in Cypress, but you don’t have to. I’ve been pretty happy with Cypress at work and for my own site.

I’m definitely open to hearing about other tools like test cafe.

Some people chimed in a while ago here.

Collapse
 
nickytonline profile image
Nick Taylor

I just read through the Test Cafe docs. Looks solid as well. It'd be good to do a comparison.

Collapse
 
shadowwarior5 profile image
Konstantin Meiklyar • Edited

Just curious, what were the limitations? Was it some functional limitation or more the concept and the way things must be done?

Collapse
 
shiling profile image
Shi Ling • Edited

🧐What's wrong with your current experience with rspec?

I could really go on about the pros and cons of different testing tools, but without knowing what's currently not working out for your team makes it hard to recommend something suitable for your requirements and preferences - because there's always a trade off when you pick a tool. 🙃

Before switching tools, I usually think care about what exactly do I want in my new tool. I try to resist jumping on to a bandwagon simple because it's popular. I knew what I disliked about Angular - it was hard to teach to junior devs - and picked Vue instead of the more popular React back in 2016. And recently, it turns out I didn't really like Elastic + Kibana for analytics because I want more flexibility with structuring data, so I'm trying out MongoDB Altas instead.

Figure out the following first:

Requirements:

  • What browsers do you want to test for now? Cypress and Puppeteer only supports Chrome.

Commitment level:

  • How much time does your team want to commit to creating and maintaining the tests? Selenium requires a lot of commitment to learn to write and setup and maintain the infrastructure.
  • Do you have time to commit to setting up pipelines to ensure tests are automatically ran? Tests that are run manually will certainly be forgotten - it's important to make sure they run automatically. Nearly all FOSS testing tools will require you to DIY integration with CI/CD. Most paid tools will offer built-in Scheduling and Monitoring capabilities.

Preference:

  • Does your team prefer BDD-style?
  • Does your team want the tests to live beside UI code in Ruby on Rails? Cabybara + RSpec is the popular choice. If your team is comfortable with Javascript, there's a lot of variety to pick from.

Other factors to consider:

  • How fast do you want the tests to run? Cypress is pretty damn fast, so is Puppeteer. I believe both of them utilises the Chrome DevTools Protocol instead of WebDriver protocol which makes it fast, but you'll lose some accuracy in terms of simulating actual interactions instead of syntactic interactions.
  • Do you want screenshots for every test and every step? Most tools require you to manually add steps to capture screenshots.
  • Do you want visual testing?

While @picocreator and I are the creators of UI-licious ourselves, we won't recommend it if it doesn't suit your style. :) It's designed to be easy for non-technical folks, whilst flexible for Javascript programmers, and supports all browsers - but some folks don't like Javascript, others want tests to run blazing fast, and plenty of folks want it free. ¯_(ツ)_/¯

Collapse
 
shiling profile image
Shi Ling • Edited

To add on, I've seen a lot of migration projects (for front-end, back-end, testing-frameworks) fail because the problem was the process and team capabilities/capacity, not the tools. I think it's important to really visualise the ideal workflow for the team, and then find the tool that fits. Finding the tool first and then fitting the team to the tool is like chopping the foot to fit the shoe (which is why I'm usually wary of overly-opinionated tools).

Collapse
 
flictuum profile image
Florent Bertrand

Thank you for your share, it really helped me to choose my testing tools ! I agree with you the first thing to do is to define what the team can do ...

Collapse
 
cristinasolana profile image
Cristina Solana

I used Cypress on a project and loved it, but I have a hard time selling it to stakeholders because it isn't cross browser.

Chrome is rarely the browser with weird bugs so testing there is considered happy path. :/

Collapse
 
shiling profile image
Shi Ling

Chrome is rarely the browser with weird bugs so testing there is considered happy path. :/

So true, since most developers use Chrome as their main dev and debugging browser - of course it looks perfect - it's a classic "it works on my machine", except replace "machine" with "browser".

I'm pretty guilty of this tendency as well. I hope to see Firefox and Edge catch up to offer a better experience for developers.

Collapse
 
cristinasolana profile image
Cristina Solana

Precisely, if I have to test manually on all supported browsers, it would be counterproductive.

Off topic, but I like Firefox Devtools and they're adding some nice new features like letting you know when css props is not having any effect, but I am very used to the Chrome experience so hard to change. :P

Collapse
 
elmuerte profile image
Michiel Hendriks

What is Firefox missing for developers?

Thread Thread
 
shiling profile image
Shi Ling

I tried to switch to FF twice this year, but while the dev tools is great, performance is noticably slower compared to Chrome on many websites.

Also, ReCaptcha is so nasty on non-Chrome browsers, I spent 5 minutes solving a challenge and still I was getting more puzzles (it's not FF's fault, Google's ReCaptcha is just so fucked up).

Thread Thread
 
elmuerte profile image
Michiel Hendriks

ReCaptcha is so nasty on non-Chrome browsers, I spent 5 minutes solving a challenge and still I was getting more puzzles

That is a privacy price I'm willing to pay.

I can't comment on performance. I have been using Firefox since 0.4. I only use chrome when needed to verify non-standard webstuff. Firefox never felt slow to me.

Thread Thread
 
shiling profile image
Shi Ling

Haha yeah... I'm spoiled by Chrome now - but these days either Chrome is slowing down or websites are getting fatter... - I might jump in to migrate to FF again.

Collapse
 
dasdaniel profile image
Daniel P 🇨🇦

Cypress is a superb choice if you're only testing using Chrome. The ability to step through your test makes writing tests easier.

If you want to test with other browsers, like IE, Edge, Firefox, Safari, and any mobile browsers, TestCafe is a way better choice (given that it is browser agnostic). They also offer the studio ($), so you can create the tests without writing it.

In short, I'd say overall the cypress experience is better, but the usefulness of TestCafe is broader.

Collapse
 
leob profile image
leob • Edited

Debuggability (as in, the ability to step through a test script using the debugger) was harder in TestCafe than I'd hoped (I think mainly due to "async"), would it be easier in Cypress? That would definitely be a big "plus" ...

Collapse
 
dasdaniel profile image
Daniel P 🇨🇦

Have a look their Toggling + Debugging video

youtu.be/ofrGbDUuEcQ?t=260

You can see that when the assertion fails, it gives you an interface to see the current state of the testing script. Hovering over the failed assertion shows where the failure happened.

In TestCafe, you can use --debug-on-fail flag or debugOnFail:true in config during test creation/debug. This will prevent the test from ending on error and is quite helpful, but the ability for cypress to step through the test after the failure makes debugging test easier.

Collapse
 
sduduzog profile image
Sdu

TestCafe runs as a node tool, which allows me to create helpers that can prepare and cleanup test data from the database as almost how a backend server would. Cypress is loaded and runs on a browser so to do the same you either add endpoints to your api just for testing or spawn up a separate testing server.

Collapse
 
ecyrbe profile image
ecyrbe • Edited

Here at work we tried cypress... and after running into it's limitations, we switched to testcafe.
Testcafe is browser agnostic (works on all of them even esoteric ones thanks to it's proxy impl).
Cypress is overrated

Collapse
 
leob profile image
leob

What are the limitations of Cypress, apart from it not being cross-browser? I'm asking because I've used TestCafe for a bit, but I'm curious about Cypress ...

Collapse
 
sesamechicken profile image
sesamechicken

Speaking from my experience when creating a POC, we couldn't use it because

  • it's Chrome/Electron only
  • each test is bound to a single origin
  • no support for multiple tabs

I will say, it was easy to use, write tests, and it was awesome to see its output. It really would've made developers hate functional tests a lot less. ymmv

Thread Thread
 
leob profile image
leob • Edited

Right ... point 1 (Chrome only) is a known issue, if you need cross browser testing then it's a no-go. Point 2 and 3 are indeed documented in their official docs:

docs.cypress.io/guides/references/...

I guess there's a large number of projects where these limitations are not a showstopper, but if you do need these particular features, well yes then Cypress is indeed not an option.

TestCafe with its "proxy" architecture is probably just a bit more flexible, OTOH for a very large percentage of projects you may not need that flexibility ... ymmv indeed.

Collapse
 
popescunsergiu profile image
Sergiu Popescu

Why does plain Selenium isn’t suddenly good enough anymore? Remember that it is the most used, support can be found everywhere and I can speak of a lot more reasons why to keep using it.
Neither Cypress, nor TestCafe give me reasons to use them. More that this, both gave me reasons not to. Tests have to be as close as possible to the real life usage of the app. Do cypress or testcafe help you build such tests? Or both help you build fast, green tests?

Collapse
 
aleccool213 profile image
Alec Brunelle

I have used both extensively and the dev experience of Cypress is about one million times better. Very good documentation and modern developer experience where Selenium is the opposite. Most docs for Selenium are in Java and finding the Javascript api's wastes time. I usually had to use Stackoverflow for documentation on Selenium where I always used the official docs for Cypress.

Collapse
 
popescunsergiu profile image
Sergiu Popescu

I think this is exactly what I said, build fast, green tests. Try to do some database manipulation in Cypress before running your tests. I found it difficult comparing with nodejs. What happens if you need extensive grouping and need multiple layers of test hooks? Support here is not that great. Try to explain to stakeholders the test coverage, maybe linking the 'it' or the 'describe' functions to requirements as it seems that more often they are written in Gherkin.
Don't get me wrong, Cypress is wonderful for max 6 months projects but if you want to go enterprise and need environment matrix support the I don't see Cypress as a good choice.
I always encourage the use of the right tool for the right job and sometimes Cypress is good, other times Selenium, or paid tools, most of them build upon Selenium.

Have anyone wonder why most of the great paid testing tools are built on Selenium. I don't think it is because Selenium is not good.

Collapse
 
mintzworld profile image
Michael Mintz

If there are Python developers on your team, you may want to check out SeleniumBase: github.com/seleniumbase/SeleniumBase
It’s a framework that combines pytest and Selenium in a simplified, but very powerful way. Like some other frameworks, it has smart waiting built-in, which means the framework will wait for page elements to fully load and be interactive before performing Selenium actions on them. Includes a lot of useful built-in methods such as assert_no_404_errors() and assert_no_js_errors(). Lots of examples from the GitHub page. Multiple browsers are supported. Proxy servers are supported. Easily connect to a Selenium Grid. Includes JavaScript injectors for building on top of websites. Screenshots, test reports, ad-blocking, visual testing, and more.

Collapse
 
scriptmunkee profile image
Ken Simeon

I agree with @shiling, that you have decide what you're trying to accomplish overall. There is nothing wrong with RSpec and your current use to validate your base Rails infrastructure [ yes, I took a quick peak ].

I can only guess that you're looking to expand your functional testing coverage with UI based testing [ not truly E2E testing ]. Its definitely helpful for when you're changing foundational underlining code that shouldn't impact the user experience or UI components. However, as its been pointed out the intentions of the UI testing needs to be defined.

Now I can't give you insight into Cypress vs TestCafe, but I can give you my thoughts or insights for adopting GUI automation from a QE perspective.

  • If the UI test creation audience is the developers and not a QE, then keep the test framework & language close to the daily used language(s) (easier adoption)

  • Be definitive about separating automated test focus [unit tests, API tests & functional UI tests].

  • UI test should validate/assert behavior and/or data interactions.

  • Treat UI tests like a unit test. They should run independently. They should not rely on another tests success or failure.

  • When UI tests are independent, all necessary data to complete a test must be seeded, created or found in the test environment [ key for a successful CI/CD/CT environment ].

  • True UI E2E tests should only be built in special cases and run during integration or before a release.

Collapse
 
pavelloz profile image
Paweł Kowalski

I prefer TestCafe.

Here are some of the reasons: dev.to/pavelloz/comment/ggn8

Collapse
 
andrewbrown profile image
Andrew Brown 🇨🇦 • Edited

I guess the question is what is the gain? Convenience?

Doesn't a Selenium-based driver ensure you can test cross-browser and since this is an open-source platform where we need to be inclusive to browser and platform choice that we'd choose Selenium which is going to give us that coverage?

Is Capyabara + Cucumber really that bad? How slow would it really be? I have a project with 40 features and it takes 1-2mins to run.

I would say before introducing and E2E tool into the codebase we get specs moved to where Rails developers expect them to be. I have little problem working with the codebase quirks but let's ease it for juniors.

Collapse
 
swarupkm profile image
Swarup Kumar Mahapatra

Capybara + cucumber. In a selenium grid docker setup.
Parallel_cucumber to parallelize the tests .
Thats 😍 for me.
I have run 420 tests accross 20 parallel threads , and complete tests under 30 minutes

Collapse
 
arturoaviles profile image
Arturo Avilés

If you need to test iframes, selenium is the best option to work with 👍🏼 I have tried using Cypress, Katalon Studio, etc. but I got stuck because they weren't compatible with iframes in some way or another. Maybe an option would be to use Puppeteer but it only uses Chrome.

Collapse
 
xvier profile image
Xavier

A colleague of my wrote 2 blogs comparing Cypress and Testcafe.

Atm I used Cypress at multiple clients and for our use cases it worked. But I can understand that the limitations of 1 browser and no cloud testing can be a big limitation for people to start using Cypress.

Here is the first part of the blog:

xebia.com/blog/cypress-and-testcaf...

Collapse
 
destro_mas profile image
Shahjada Talukdar • Edited

I gave a Talk about Cypress in Berlin, Germany. Please have a look. It has slides but not the code samples of demo.

Collapse
 
angrykoala profile image
angrykoala • Edited

While facing a similar situation and trying several tools, I ended up making github.com/angrykoala/wendigo with the intention of making frontend+e2e tests as simple as possible

It is basically a wrapper of Puppeteer (with all the goodness and bad of Puppeteer), it's syntax is similar to what you would find with cypress but way less opinionated over how to write your tests and what other tools you may use (which is the main downside I found with Cypress). Not sure if it may help you but you may want to give it a look as an alternative

Some comments may only be visible to logged-in visitors. Sign in to view all comments.