I'm not familiar with Cypress but it looks like it brings tools together to provide a complete solution. Playwright/Puppeteer only provide communication to the browser.
Is Cypress working for you? What issues do you have with it?
My experience lies with C# and selenium. This past couple weeks I've been diving into Puppeteer (Playwright is a fork). We are utilizing it with jest and it is not anywhere near as stable or fast as our selenium solution. This isn't about the tools though, we need to put in the work to achieve the same standards.
Also, we aren't writing end to end tests. At some point I may identify a place this in need of E2E, but I look to avoid that and have appropriate mocks/stubs in place.
So don't look at changing tools, look at changing test strategy.
I'm a fan of Open Source and have a growing interest in serverless and edge computing. I'm not a big fan of spiders, but they're doing good work eating bugs. I also stream on Twitch.
Where I'm currently at, DEV/Forem, we're still looking into an E2E tool. We currently use Capybara with Rspec, but it is mainly integration tests from what I've seen.
I'm just looking at other tools still as it is an open source project. Here's the issue I created a couple of years ago. Funnily enough as I read through the issue again, I can see I mentioned Playwright when I started at DEV/Forem.
Story - Consider Adding End to End (E2E) Testing to the Deployment Pipeline
@benhalpern @maestromac
@Zhao-Andy @jessleenyc et all.
Just wanted to open a discussion here about E2E testing.
There's a lot going on on the site. The backend has lots of unit tests, the new front-end stuff has unit tests and snapshot testing, but there is no E2E testing in place.
When I worked at McAfee, it was essential to have E2E testing in place because we were working on a password browser extension. We had to be certain the UI functioned all the time. In the case of a web site, it's a good idea as well, but a lot of times it's omitted due to lack of resources/time. Not good excuses, but that's the reality I've lived through at other jobs. There's also a burden of maintaining them.
Having said that, they help you spot regressions fairly quickly and it allows you to test full workflows, e.g. on-boarding. In terms or resources, I think we'll have those once the code is fully open sourced. I have a feeling a few people will want to contribute to the code base. 😉
I think it's a win win honestly. The site will be more stable because of the E2E tests and there will be so many tests to do that it will be an easy way for new contributors to get some PRs in (provided we onboard properly).
If we decide to go ahead with this, we'll need to decide on an E2E testing framework. I'm personally leaning towards Cypress, but let's make this part of the discussion.
Yeah let's do this. I haven't seen any reason to not go with Cypress. It's definitely been on my list of things to try and haven't seen anything better.
Rails has its own concept of E2E testing which we make some use of but it's been such a pain in the ass I've never had the patience to be good at this part. I'll leave this topic open for anyone to argue that we should stick into this world as opposed to Cypress.
Either way, yeah I'm in agreement with the approach to lead to stability, as long as we don't get too heavy-handed to the point where we need to change 10 files for every change. Let's try to walk the line the right way.
This is likely due to my misunderstanding of E2E. I generally find E2E implementations tend to focus only on one end, which is fine when you already have all the integration testing that validate those ends.
But even then the E2E testing is executed in the wrong place. E2E is a validation of the system setup. All of the efforts goes into verifying lower environments. Being test environments they are "broken" to test a situation. So integration tests fail and everyone is trying to find that one person who can put it back together.
But with a good suite of contract tests in place you focus on the real issue, components aren't connected together correctly. Anyway I ramble.
I'm not familiar with Cypress but it looks like it brings tools together to provide a complete solution. Playwright/Puppeteer only provide communication to the browser.
Is Cypress working for you? What issues do you have with it?
My experience lies with C# and selenium. This past couple weeks I've been diving into Puppeteer (Playwright is a fork). We are utilizing it with jest and it is not anywhere near as stable or fast as our selenium solution. This isn't about the tools though, we need to put in the work to achieve the same standards.
Also, we aren't writing end to end tests. At some point I may identify a place this in need of E2E, but I look to avoid that and have appropriate mocks/stubs in place.
So don't look at changing tools, look at changing test strategy.
Cypress works well.
Where I'm currently at, DEV/Forem, we're still looking into an E2E tool. We currently use Capybara with Rspec, but it is mainly integration tests from what I've seen.
I'm just looking at other tools still as it is an open source project. Here's the issue I created a couple of years ago. Funnily enough as I read through the issue again, I can see I mentioned Playwright when I started at DEV/Forem.
@nickytonline commented on Thu Jul 19 2018
Story - Consider Adding End to End (E2E) Testing to the Deployment Pipeline
@benhalpern @maestromac @Zhao-Andy @jessleenyc et all.
Just wanted to open a discussion here about E2E testing.
There's a lot going on on the site. The backend has lots of unit tests, the new front-end stuff has unit tests and snapshot testing, but there is no E2E testing in place.
When I worked at McAfee, it was essential to have E2E testing in place because we were working on a password browser extension. We had to be certain the UI functioned all the time. In the case of a web site, it's a good idea as well, but a lot of times it's omitted due to lack of resources/time. Not good excuses, but that's the reality I've lived through at other jobs. There's also a burden of maintaining them.
Having said that, they help you spot regressions fairly quickly and it allows you to test full workflows, e.g. on-boarding. In terms or resources, I think we'll have those once the code is fully open sourced. I have a feeling a few people will want to contribute to the code base.😉
I think it's a win win honestly. The site will be more stable because of the E2E tests and there will be so many tests to do that it will be an easy way for new contributors to get some PRs in (provided we onboard properly).
If we decide to go ahead with this, we'll need to decide on an E2E testing framework. I'm personally leaning towards Cypress, but let's make this part of the discussion.
@benhalpern commented on Fri Jul 20 2018
Yeah let's do this. I haven't seen any reason to not go with Cypress. It's definitely been on my list of things to try and haven't seen anything better.
Rails has its own concept of E2E testing which we make some use of but it's been such a pain in the ass I've never had the patience to be good at this part. I'll leave this topic open for anyone to argue that we should stick into this world as opposed to Cypress.
Either way, yeah I'm in agreement with the approach to lead to stability, as long as we don't get too heavy-handed to the point where we need to change 10 files for every change. Let's try to walk the line the right way.
@nickytonline commented on Sun Jul 22 2018
Also, it currently only supports Chrome, but FireFox support is on it's way.
@nickytonline commented on Thu Aug 16 2018
@maestromac , can you migrate this issue over? I'll put up my PR in my branch later tonight or tomorrow. Thanks.
This is likely due to my misunderstanding of E2E. I generally find E2E implementations tend to focus only on one end, which is fine when you already have all the integration testing that validate those ends.
But even then the E2E testing is executed in the wrong place. E2E is a validation of the system setup. All of the efforts goes into verifying lower environments. Being test environments they are "broken" to test a situation. So integration tests fail and everyone is trying to find that one person who can put it back together.
But with a good suite of contract tests in place you focus on the real issue, components aren't connected together correctly. Anyway I ramble.
Eliminating Environments
Jesse Phillips ・ Sep 7 ・ 3 min read