This post was originally published on our blog.
Developers are a fussy bunch. We sweat on the tiniest details and every aspect of our product has to be just right. We don’t like entrusting our precious designs to anyone outside our inner circle.
Yet we’re increasing delegating key quality assurance (QA) tasks to robots. The market for automated testing products is expected to be worth $20 billion by 2023 - three times as much as now. The ever-increasing complexity of software programmes, which have grown from a rudimentary series of command lines to a potentially infinite number of variables, have forced us to seek help from artificial intelligence. What’s more, the growing influence of agile and DevOps means we’re in a permanent sprint, and we need to speed up the QA process wherever we can.
Automation testing is much faster than its human equivalent, and yields crucial analytics. It lets us run tests 24-7, even several at once. Perhaps most crucially, it enables us, as developers, to focus on the creative stuff, freeing us up for what’s most important. Unfortunately there are a baffling number of automated mobile app testing tools to choose from, and each one comes with its own bullish marketing literature, telling you that this product is more reliable than any other. Given the baffling amount of jargon and technical-speak involved, it’s hard to separate the real from the spiel.
Since our parent company Mobile Jazz started out in 2012, we’ve tested our apps thousands of times, and we’ve put plenty of testing tools through their paces. So we wanted to compile a list of our five favourite automated mobile testing tools, to give fellow developers the benefit of our long (and often painful) experience.
Just a quick reminder: this list targets Android app automation testing tools. We’ll compile a list of the best options for iOS in the not-too-distant future, so iPhone specialists should stay tuned!
1) Appium
What it offers:
- Android, iOS, Windows, and Firefox OS platforms
- Open source
- No need to recompile app
- Supports cross-platform application testing
How much does it cost?
It’s free. Completely free
What are its biggest strengths?
- Versatility across languages and testing systems
- Huge open-source community for advice and support
A perennial favourite of app testers everywhere, the Appium tool is completely free and it’s available through GitHub, so there’s no complex installation required. The product is provided through open source; in fact the community that built it, Sauce Labs, is pretty active so it offers a gateway to a wealth of advice and support.
Another reason our QA specialists love Appium automated testing is its versatility. The software works with any testing system and supports a number of dialects, including Ruby, Java, PHP, Node and Python. It’s also pretty low-maintenance: standard robotisation APIs mean there’s no need to recompile your application in any way. The Appium android testing tool doesn’t expect anything to be introduced on the gadget, and it can be set up on an alternate stage easily.
On the flip side, tests can take a while because they’re being conducted by a remote webdriver, and the reports can lack detail. If you’re setting up Appium locally, your team will have to download, install and configure the environment, and you’ll need to connect a local device - which can be a hassle. Plus, because it’s community-supported, it can be slower to pick up the latest OS developments than rivals.
Overall though, we think Appium mobile testing performs strongly for all the key criteria, and its advantages are applicable to a wide range of requirements.
2) Robotium
What it offers:
- Android
- Open source
- Supports both native and hybrid apps
How much does it cost?
It’s free
What are it's biggest strengths?
- It’s quick
- It can do several different tests at once
- The tests are robust
A giant of the mobile-friendly test sector, Robotium is renowned for the speed of its experimental cases, so it’s great if you’re trying to conduct QA in an agile testing environment. Another major plus is that it needs very little information to perform effectively, so you don’t require a forensic knowledge of the application.
Robotium android automated testing can be performed on several different devices simultaneously, and the tests can run on either an emulator or the real thing. They are also renowned for being extremely robust, because Robotium recognises UI elements by their internal code reference; if you’ve built your product using Java, you can combine Robotium with the JUnit test framework for even stronger results. Yet perhaps the coolest thing of all is Robotium can simulate ordinary user behaviour, like touching, clicking and typing text.
However, be warned. Although the test themselves are done practically before you can say ‘all clear’, it can take a lot of time to get the tests ready, because of the usage of the program source code. It can’t bolt and open a cellphone or tablet, and it doesn’t provide play or record functionality.
To be honest, however, this is a really, really good alternative and would come top of many people’s android app development tools list. And given the price (well, the lack of a price) it’s a major bargain for small development teams with shallow pockets.
3) Kobiton
What it offers:
- Cloud-based system built via open source
- Can run both manual and automated tests
- Movement logs, charges, screen captures and metadata are all automatically created
How much does it cost?
It’s free, although you can upgrade for between $50 and $300 per month.
What are its biggest strengths?
- It allows you to test on real devices
- It lets you do manual as well as automated tests
- Continuous updates mean users are always ahead of the curve
A cloud-based solution provided over Appium’s open-source system, Kobiton allows you to use real phones using its awesome feature. You can perform both mobile app testing and mobile web test runs, and flit between manual and automated options - so you can always go back to the old-fashioned hands-on method if you feel the urge. You can also build your own customised test cloud using Kobiton’s Device Lab Management feature.
The mobile android testing kit is continuously updated, so you’ll always have the most recent equipment and OS iterations for each mobile device testing session, and you can test crosswise over gadgets without content alterations. Movement logs, charges, screen captures and metadata are all automatically created, so you’ll have all the metrics you could conceivably need.
There are a few drawbacks compared with other android test options. You may get the dreaded timeout error with certain commands, and video-streaming is not supported if you’re using IE 11 and Edge browsers. Also, certain screen settings can’t be changed as they might potentially affect Kobiton’s devices.
These are just minor quibbles though. As early adopters who love geeking out over new software updates, we’re huge fans.
4) Ranorex
What it offers:
- Android and iOS
- Cross-browser and cross-device testing for iOS as well as Android devices
- Simple integration with CI servers, issue tracking tools, etc
- You can also integrate Jira with Ranorex Studio
- Ranorex Recorder, Code Editor, Automation helpers, and other useful tools
How much does it cost?
Free demo. Upgrade from €2,290 to €3,990. See more.
What are its biggest strengths?
- It’s really simple to use - you don’t need to be a veteran tester or programmer
Of all the automated testing tools on our list, none of them is more simple or adaptable than this one. If you’re not from a programming background or you’ve never done automated software testing before, Ranorex lets you run your test without a script. It easily integrates with other testing tools such as TeamCity and nCover, and it comes with robust debugging capabilities.
Ranorex offers full cross-device testing, so you can try out websites and desktop applications as well as mobile apps, and it provides cross-browser testing functionality for Chrome, Firefox, Safari and Microsoft Edge amongst others.
Drawbacks? Well, unlike some of the other automated android testing tools on our list, there’s no open source capability. To run tests, you need to connect your mobile devices via Wifi or USB, which can be tricky in certain locations, and you must install the Ranorex control app on each mobile device before you run a test on it. One final thing to note is that, unlike many rival testing tools, Ranorex does not have a device viewer when recording, so you’ll need access to the mobile device.
Overall, then, it has some limitations compared to its rivals, but if you’re a small team without much specialist testing expertise, Ranorex is a superb option.
5) Calabash
What it offers:
- Access to the Xamarin Test Cloud
- Cucumber automation
- Open-source development
How much does it cost?
Zero. Null. 零 (that means ‘0’ in Chinese). Basically, it’s free.
What are its biggest strengths?
- It lets you access the Xamarin Test Cloud and its hundreds of devices
- It supports plenty of different programming languages
- You can write your tests in plain English using Cucumber automation
If you’re not heard of the Xamarin Test Cloud before, well, it’s possibly the best thing invented since the Game Boy. It gives users access to over 1,000 different devices in the cloud, allowing them to try out their product on practically every smartphone ever conceived.
The Calabash automated android testing tool, an open-source product developed and maintained by Xamarin, harnesses the power of this technology. Users are invited to write their own bespoke test cases using Calabash’s libraries, which support a Ruby-based framework. Then, they can put them into practice on the Xamarin Test Cloud.
Yet, for many people an even bigger advantage of Calabash is its support for Cucumber, an offshoot of Ruby which enables you to write automated tests in plain English rather than arcane code. If you don’t have specialist QA staff in your team, the Cucumber automation is a serious boost.
There are, of course, some limitations. Calabash only supports Ruby, so if you’re used to working with alternatives like Java or Python, it probably isn’t for you. The tests can also be time-consuming and it can be a bit unwieldy and frustrating on the command line side.
But these are minor quibbles. If you’re light on QA expertise and don’t mind spending a bit of time setting the tests up, we’d definitely recommend Qalabash.
More Options
This list is by no means exhaustive. There are several potential alternatives to pick from, including UIAutomator, MonkeyRunner (our favourite name of all the automated android testing tools in the world) and Selenium. They may not have made the cut this time, but they’re still damn fine options.
And there’s one more we’d like to mention…
Bugfender
Sorry for the tiny bit of self-promotion, but well, we figure it’s not hurting anyone down here.
Bugfender is a cross-platform tool which goes beyond normal crash reporting tools; it is more like a regular beta testing tool, one which helps you to identify any bug on any device and gives you a panoramic view of both your application and your user base.
It’s still very much a product in development, we’re happy to admit that. It’s not available for web, Windows or Apple’s tvOS as yet. But we’re proud of the functionality it offers mobile development teams. Bugfender is specifically designed to confront the problems we see on a daily basis as developers - most notably Android fragmentation, the exponential proliferation of new devices which means we now have to consider thousands of different smartphones when designing our apps and websites.
Bugfender does not require any physical installation - it can be simply loaded onto any user’s device, even if it’s an obscure model you’ve never heard of. It logs 24-7, so you don’t just get crash reports - you get a forensic view under the hood of your application, even when things are running smoothly, and you get a breakdown of all the devices using your product, which is great for customer service.
The product is free, although we’ve also got an upgrade available for €49. If you’d like to find out more, go to Bugfender’s website.
This post was originally published on our blog.
Top comments (0)