This is more a question than an article. Is it worth spending time on automated testing on a product you've not yet launched? Especially when you're a small team, or even a single person on your own.
By this, I don't mean Unit Tests, but the automated acceptance tests - both at the API and the UI level.
On the one hand, anything that takes time away from feature development means it takes longer to get to market, and this in itself is a bad thing. Longer to market means more chance of getting beaten to the punch, more time to get customers, etc. There's also the fact that UI tests are generally quite brittle, and even API tests pre-launch are likely to be subject to change much more often.
On the other hand, if you don't write the tests as you go, why will you go back and write them later? Some technical debt you need to pay back, others you can keep rationalising away. And a lack of automated tests means more time doing manual tests, which ends up with lower confidence in things working right.
So - is it worth spending the time writing automated tests before launch?
Top comments (16)
It's 2017 and people are STILL asking this question? Do you enjoy spinning up your application to test the entire thing just to ensure your small patch didn't break anything else? Are you confident you didn't miss anything, or as stevebriley mentions - are you happy for your customers to find obvious bugs?
FFS - Use CI and TEST IT!
Yes, Dale! I'm a tester and many people still think it's a great idea to hack together and get it live ASAP. Then having a meeting with the business and the PMs 2 days later wondering why it's broken!
I am going to go ahead and say no, you should not write these kinds of UI tests before reaching market. In my experience, things change a lot in this regard before, during, and after initial launch of a product, and the overhead of changing the tests with every change is not going to be worth it.
Software development in the wild is all about tradeoffs and risk management, and the risks of creating too much overhead outweigh the risks of breaking the UI a few times. This post touches on this subject well.
I'd say focus on good monitoring practices early on as opposed to restrictive testing. It will take a while until the mushy parts of UX start to solidify and you need to take on the risks of slightly more brittle software in order to mitigate the risks of process overhead.
I'll echo exactly what Ben has said. Automated tests are great, when there are a lot of people working on something, or when it is a big company and there's a codebase that has matured. If you're building something with a couple of people or by yourself, I'd say don't focus on tests. Add some tests for any business critical parts that would be disastrous if it breaks, or as you're fixing bugs maybe add some tests (test to prove it's broken, and validate it is fixed with your change).
I say yes to the API level tests. This will ensure you design a system that is testable from the ground up, and not have to go through the pain of trying to add them in later when you have already written a lot of code.
Testing through the UI, the value dramatically drops off. In my experience UI testing is very brittle. Any change in the UI will break the tests, or even stupid things like a window that popped up unexpectedly. I agree with Ben on this. The early days of the project the UI might be changing every few hours, you don't want to waste time rewriting the UI tests again and again.
Agreed. Even testing guru uncle bob Matin sais you should always write intergration tests, but never have them run from the ui. The ui just changes to fast, which makes the tests verry flaky and defeats the purpose of testing for stability.
For myself, I think API level tests are absolutely worth it, but I'm not sure UI tests are too early on in the process.
API tests can be relatively easily changed if you decide that the API design is wrong. UI tests are inherently more brittle and are more expensive to rewrite if/when the UI does change. They're also slower to run and more difficult to get set up correctly in the first place.
Omg. Yes automated tests. If I was a purchaser of your product, and I wanted to run my fortune 500 business off of it in a mission critical way, how are you going to prove that your product didn't suffer any functionality reversions between versions?
How are you going to prove it's interaction with the other systems, that your software ties into, actually works?
Why do you believe that fixing bugs later in the SDLC, will somehow save you time and money, over early detection of those same issues?
To me, not having automated tests, is the equivalent of technical debt. You have no way to prove that your software runs reliably, or even runs correctly. It also leads to poor design.
Making the assumption that manually testing is going to save you time is the most insane thing I have ever heard of.
What evidence do you have that you are so infallible, that you won't waste development cycles by manually testing?
What evidence do you have that you will even manually test consistently?
What evidence do you have that your co-workers will manually test things consistently as well?
Also, how would you expect to convey these multiple manual tests to others?
Are you comfortable with your customers finding bugs in your software?
Will that cause reputation damage to your software and business/brand?
... To me, those are the real questions that should be asked.
Tests will allow you to iterate more quickly as you get user feedback so I would want them. UI tests can be brittle but are very important to ensure you don't break the UX. Try using approval tests to reduce the cost of UI testing and still alert you to changes.
Better than an answer I'm going to give you another perspective. Testing is not your goal, it's just a tool, your specific project has risks and testing is the perfect tool to mitigate some risks.
So the question is, how risky is to build a single screen app with little UI compared to an app with a nice and friendly UI flow? When your project is the first app, then the risks after skipping end to end UI tests are mostly the same, when you build the second one, you know the risks of having problems with different web browsers, Android quirks or phone models are very real and not small, so in this case end to end UI tests do mitigate those risks very effectively.
A simple analogy, suppose you work in a technology demo without external dependencies and no plan to deploy an application. In this case, there is no shipping risk that a Continuous Integration environment could mitigate, so you can spend time setting up CI, but it's obviously not a risk.
Finally, you might not know the answer as to where is your risk, in that case, ask the business how much they want to invest in UIs now and in the future.
I don't think you can ever have a blanket rule like this. "launch" can mean many things. If you're cranking out a 4 week MVP, then automated tests may not make sense because as others said: things can change a lot.
However, if your MVP simply has to be elaborate, then an automated test could help you during development before launch. Automated tests aren't just for ensuring users don't see bugs, they are for developers to save time.
Which leads to the last point: how quickly can you build those tests? Some kind of UI tests I can build in minutes. Some will take me days. The ones that take me minutes will most likely save me more time within a few weeks, maybe even a few days. The tests that take me days to build won't do that.
I would say that API level tests are mandatory. You want your app to be rock-solid from the start, not crashing on stupid 500 errors you could have easily identified with good tests. Avoiding technical debt from the start is also a really good idea if your app actually kick off very quickly... you will have little time to go back on your first decisions. It doesn't take much time with modern CI tools to setup a proper backend testing infrastructure and totally worth it.
On the other side, i've had some very hard time maintaining a good infrastructure for end-to-end UI tests. These are really subject to change, especially from the start. I would prefer going with solid unit tests on that side.
I've had experience in both and I vote for tests.
Test the big things, test the things you know aren't going to change next week. It's kind of tedious at first, but where the tests make up for it is once your product is done, and a new version of x dependency is out, you'll know quickly whether something major broke or not. Also a fan of CI like someone else mentioned, and having that test for you.