As developers, we have the responsibility to provide our users with functional applications.
Each year, I explore all the relevant solutions for automated testing and rank them based on a list of requirements.
Disclaimer
I do not discriminate against open source or proprietary technologies.
I do wish all the solutions were free and open source, but one of my requirements is for the solution to include a cross-browser cloud.
Because some of our users are using Safari or Internet Explorer 11.
If you're preparing to type a comment like this:
But did you try that E2E-library-that-only-works-for-Chrome? 🤓
Please stop and check if it fulfils most of the listed requirements.
If it does, please tell us about it in a comment.
I personally don't believe in open source solutions, backed up by huge marketing budgets spent on bashing Selenium.
| Requirement | Endtest | BrowserStack | Sauce Labs |
|---|---|---|---|
| Works for Web Applications | ✅ | ✅ | ✅ |
| Works for Mobile Applications | ✅ | ✅ | ✅ |
| Solid Documentation | ✅ | ✅ | ✅ |
| Cloud with browsers on Windows machines | ✅ | ✅ | ✅ |
| Cloud with browsers on macOS machines | ✅ | ✅ | ✅ |
| Codeless Technology | ✅ | 🔲 | 🔲 |
| API | ✅ | ✅ | ✅ |
| Scheduler | ✅ | 🔲 | 🔲 |
| Useful Customer Support | ✅ | 🔲 | ✅ |
| Email Testing | ✅ | 🔲 | 🔲 |
| Geolocation | ✅ | ✅ | ✅ |
| Video recording of test executions | ✅ | ✅ | ✅ |
1. Endtest
A cloud platform that allows you to create, manage and run Automated Tests for Web Applications and Native Mobile Apps (Android & iOS), without having to write any code.
Lots of nice features, such as:
• Cross-browser grid, running on Windows and macOS machines
• Codeless Editor for Automated Tests
• Support for Web Applications
• Support for both native and hybrid Android and iOS apps
• Video recordings for test executions
• Detailed logs
• Chrome extension to record web tests
• Element Inspector for mobile apps
• Screenshot comparison
• Data-driven testing with CSV files
• Geolocation
• Email, Slack and Webhook notifications
• If Statements and Loops
• Variables and re-usable components
• Support for file uploads in the tests
• An Endtest API, for integration with CI/CD systems
• Advanced Assertions
• Endtest Mailbox, for testing emails
• Self-healing tests
This is what it looks like when you want to execute a test:
And these are the detailed logs from a test execution:
Their Documentation is fantastic. Lots of examples and videos.
I got quick responses from the Endtest Support, which is constantly praised in the reviews I found online.
Netflix uses this platform to test their web apps. It's even listed as a Required Skill for some of their jobs.
Clearly, Endtest is the winner.
2. BrowserStack
BrowserStack is a cloud platform that offers you a list of browsers and mobile devices to execute your Selenium and Appium tests.
Features:
• Cross-browser grid, running on Windows and macOS machines
• Mobile device lab with emulators and real devices
• Video recordings for test executions
• Detailed logs
• Geolocation
• BrowserStack API
The User Interface is definitely intuitive:
The Documentation from BrowserStack is well structured and offers numerous examples.
What I didn't like about it is the amount of effort it takes to make it work.
It doesn't offer any out-of-the-box features.
The Support from BrowserStack could be better. They do not have a chat and it takes a few hours to get a response.
3. Sauce Labs
Sauce Labs is very similar to BrowserStack, also a cloud platform that offers you a list of browsers and mobile devices to execute your Selenium and Appium tests.
With similar features:
• Cross-browser grid, running on Windows and macOS machines
• Mobile device lab with emulators and real devices
• Video recordings for test executions
• Detailed logs
• Geolocation
• Sauce Labs API
This is what it looks like:
It's difficult to navigate through the Documentation from Sauce Labs.
You have to spend some time digging around.
For example, let's say you want to execute your Appium tests on their mobile devices.
The ID for each device is required, but you won't find any place that contains the list of devices and their IDs.
Their API feels a bit incomplete.
I was surprised to find out that you can't upload the latest build for your mobile app with the API, you have to do it manually through the User Interface.
That's a shame, because both Endtest and BrowserStack offer that functionality.
Our experience with their Customer Support was pleasant.
Top comments (10)
I think the title is misleading and click-baiting, you have mentioned only cloud providers for running tests.
The best tool depends on the project and the team. In my situation I would not recommend or use any of those mentioned in 2020.
I agree, and would add that this is not a fair comparison as both Browserstack and Saucelabs offer no test creation tools. Endtest is more akin to offerings like testcraft.io and Microfocus or Smartbears suite of tools. Overall the article reads more like an advert for Endtest than a balanced discussion.
Hi Dan,
I see you just created your account here on DEV Community.
We had to find a solution for both Web and Mobile that would provide a cross-browser cloud.
We specifically needed browsers on Windows and Mac OS.
That was one of the most important requirements and it was not optional.
We would have been ok with creating our own tests with Selenium and Appium, that is why we did not exclude BrowserStack and Sauce Labs.
The "codeless" requirement was optional.
We did take TestCraft into consideration.
But it got disqualified due to the following issues:
• The lack of transparent pricing
• No support for mobile apps
• No clear answer if browsers are actually on Linux machines
• Customer Support was slow to reply
We also tried the other solutions that you mentioned, but they were also disqualified.
We researched over 30 solutions.
Without the information you have just added the article appears to suggest you only considered the 3 mentioned. If you had begun with an explanation of the process and that out of the many available these are your top three, I think people would have responded more positively.
BTW: I have just created my account, because this is the first article I felt I wanted to comment on. Why did you feel the need to mention that?
I have mentioned only solutions that also provide cross-browser clouds.
That was one of the items from my list of requirements.
What can I do if part of our users are using Internet Explorer 11 and Safari?
Platform is not everything, for instance I highly doubt you'll even touch iframes or shadow dom with endtest. Did you try to test a table with 1000 elements and virtual scroll ? What about a table in a canvas ? Maybe test that the data in a table is the same as in the database or backend or that formatting and styling for instance is valid for every cell. How do you do that if you can't code in a real language ?
Would endtest still be the best tool ? Most likely it wouldn't even be able to read anything on the page in any enterprise app.
I checked your other articles, you clearly plug in endtest in every article. In my opinion endtest is basically useless for anything else than static or very simple apps, prove me wrong.
Not trying to troll, but I don't like it if my favourite platform is getting full of promotion without merits.
Hi Istvan,
Those are some good questions.
Let's take a look at the Documentation.
You asked about iframes and Shadow DOM.
As you can see in the Finding elements in Web Applications chapter, this tool supports both.
You just need to use the "Switch to iframe" and the "Switch to Shadow DOM" actions.
You asked about canvas elements.
As you can see in the Dealing with Canvas Elements chapter, this tool can deal with Canvas elements.
You asked about connecting to a database and performing a SQL query.
As you can see in the Execute SQL queries chapter, you can easily connect to a database and perform SQL queries.
You asked about testing the styling.
As you can see from the Assertions for Web Tests chapter, you can add different assertions, including Element Contains CSS Class, Element Contains Attribute, etc.
And to make sure that it looks as expected, you can add Screenshot Comparison steps.
And if you want to verify a long list of elements, you can just add a loop.
You can even add a CSV file and loop through the rows and generate variables from each row and use them in a test.
I never make statements about a library or a tool before reading the Documentation.
You should do the same.
Agreed. Title cleary misleading as nothing about cloud based in title.
Hello, recently I have published card decks for young people to learn Javascript and Python. Would you be able to share the information? summonthejson.com/products/summon-...
This is literally just an ad for endtest.