This is more a question than an article. Is it worth spending time on automated testing on a product you've not yet launched? Especially when you'r...
For further actions, you may consider blocking this person and/or reporting abuse
It's 2017 and people are STILL asking this question? Do you enjoy spinning up your application to test the entire thing just to ensure your small patch didn't break anything else? Are you confident you didn't miss anything, or as stevebriley mentions - are you happy for your customers to find obvious bugs?
FFS - Use CI and TEST IT!
Yes, Dale! I'm a tester and many people still think it's a great idea to hack together and get it live ASAP. Then having a meeting with the business and the PMs 2 days later wondering why it's broken!
I am going to go ahead and say no, you should not write these kinds of UI tests before reaching market. In my experience, things change a lot in this regard before, during, and after initial launch of a product, and the overhead of changing the tests with every change is not going to be worth it.
Software development in the wild is all about tradeoffs and risk management, and the risks of creating too much overhead outweigh the risks of breaking the UI a few times. This post touches on this subject well.
I'd say focus on good monitoring practices early on as opposed to restrictive testing. It will take a while until the mushy parts of UX start to solidify and you need to take on the risks of slightly more brittle software in order to mitigate the risks of process overhead.
I'll echo exactly what Ben has said. Automated tests are great, when there are a lot of people working on something, or when it is a big company and there's a codebase that has matured. If you're building something with a couple of people or by yourself, I'd say don't focus on tests. Add some tests for any business critical parts that would be disastrous if it breaks, or as you're fixing bugs maybe add some tests (test to prove it's broken, and validate it is fixed with your change).
I say yes to the API level tests. This will ensure you design a system that is testable from the ground up, and not have to go through the pain of trying to add them in later when you have already written a lot of code.
Testing through the UI, the value dramatically drops off. In my experience UI testing is very brittle. Any change in the UI will break the tests, or even stupid things like a window that popped up unexpectedly. I agree with Ben on this. The early days of the project the UI might be changing every few hours, you don't want to waste time rewriting the UI tests again and again.
Agreed. Even testing guru uncle bob Matin sais you should always write intergration tests, but never have them run from the ui. The ui just changes to fast, which makes the tests verry flaky and defeats the purpose of testing for stability.
For myself, I think API level tests are absolutely worth it, but I'm not sure UI tests are too early on in the process.
API tests can be relatively easily changed if you decide that the API design is wrong. UI tests are inherently more brittle and are more expensive to rewrite if/when the UI does change. They're also slower to run and more difficult to get set up correctly in the first place.
Omg. Yes automated tests. If I was a purchaser of your product, and I wanted to run my fortune 500 business off of it in a mission critical way, how are you going to prove that your product didn't suffer any functionality reversions between versions?
How are you going to prove it's interaction with the other systems, that your software ties into, actually works?
Why do you believe that fixing bugs later in the SDLC, will somehow save you time and money, over early detection of those same issues?
To me, not having automated tests, is the equivalent of technical debt. You have no way to prove that your software runs reliably, or even runs correctly. It also leads to poor design.
Making the assumption that manually testing is going to save you time is the most insane thing I have ever heard of.
What evidence do you have that you are so infallible, that you won't waste development cycles by manually testing?
What evidence do you have that you will even manually test consistently?
What evidence do you have that your co-workers will manually test things consistently as well?
Also, how would you expect to convey these multiple manual tests to others?
Are you comfortable with your customers finding bugs in your software?
Will that cause reputation damage to your software and business/brand?
... To me, those are the real questions that should be asked.
Tests will allow you to iterate more quickly as you get user feedback so I would want them. UI tests can be brittle but are very important to ensure you don't break the UX. Try using approval tests to reduce the cost of UI testing and still alert you to changes.
Better than an answer I'm going to give you another perspective. Testing is not your goal, it's just a tool, your specific project has risks and testing is the perfect tool to mitigate some risks.
So the question is, how risky is to build a single screen app with little UI compared to an app with a nice and friendly UI flow? When your project is the first app, then the risks after skipping end to end UI tests are mostly the same, when you build the second one, you know the risks of having problems with different web browsers, Android quirks or phone models are very real and not small, so in this case end to end UI tests do mitigate those risks very effectively.
A simple analogy, suppose you work in a technology demo without external dependencies and no plan to deploy an application. In this case, there is no shipping risk that a Continuous Integration environment could mitigate, so you can spend time setting up CI, but it's obviously not a risk.
Finally, you might not know the answer as to where is your risk, in that case, ask the business how much they want to invest in UIs now and in the future.
I don't think you can ever have a blanket rule like this. "launch" can mean many things. If you're cranking out a 4 week MVP, then automated tests may not make sense because as others said: things can change a lot.
However, if your MVP simply has to be elaborate, then an automated test could help you during development before launch. Automated tests aren't just for ensuring users don't see bugs, they are for developers to save time.
Which leads to the last point: how quickly can you build those tests? Some kind of UI tests I can build in minutes. Some will take me days. The ones that take me minutes will most likely save me more time within a few weeks, maybe even a few days. The tests that take me days to build won't do that.
I would say that API level tests are mandatory. You want your app to be rock-solid from the start, not crashing on stupid 500 errors you could have easily identified with good tests. Avoiding technical debt from the start is also a really good idea if your app actually kick off very quickly... you will have little time to go back on your first decisions. It doesn't take much time with modern CI tools to setup a proper backend testing infrastructure and totally worth it.
On the other side, i've had some very hard time maintaining a good infrastructure for end-to-end UI tests. These are really subject to change, especially from the start. I would prefer going with solid unit tests on that side.
I've had experience in both and I vote for tests.
Test the big things, test the things you know aren't going to change next week. It's kind of tedious at first, but where the tests make up for it is once your product is done, and a new version of x dependency is out, you'll know quickly whether something major broke or not. Also a fan of CI like someone else mentioned, and having that test for you.
If I had to chose a decent compromise, I'd invest a lot in integration tests rather than units or UI specific ones. Doing so you'll get a decent coverage and confidence without dedicating too much time.
We have an e2e test that saved us a lot. It tests registering an account, the onboarding and subscription. Why? If someone can't even register in your app and you don't know it, then you are shooting yourself in the foot. It does not matter if it's mvp or advanced state.
Few people will come back to a product that they can't even start using properly. You don't need to test the whole application. Just put some safeguards to be sure that the users can at least start using your app. And if you don't have time to test, you should have monitoring like sentry.io that at least will tell you that there might be an issue.
It's relatively easy to do but need some time to setup and a bit of money depending on your case. Like using Browserstack, etc.
I think it's worth it, just identify the crucial parts and do the rest later.
I do automated tests during development - I made snaptest.io to make it quick though because it is all about speed in this initial feature creation phase.