I always assumed that unit tests are just part of the job. They are just there, and it's a no-brainer to always test every new piece of code that you add to the codebase.
Until I recently stumbled upon a guy who leads an engineering team with a radically different approach. They refused to write even a single automated test for their entire production application.
You might say, "Well, this guy is just fooling around with his 10 users."
But when I dug a little deeper into the discussion, I found that there are teams at Facebook, Twitch, and Netflix that followed a similar strategy.
So what's that all about? Should we all just stop unit testing from now on?
β Supposed Benefits of Unit Tests
You can probably find more, but these benefits are mentioned over and over again:
Unit tests help you to:
- find bugs early on before you commit them to production,
- catch bugs when your code changes,
- improve your code design,
- document how your system works.
If we assume you are not writing garbage unit tests, these benefits are all reasonable. But they come at a cost.
π° The Cost of Unit Testing
Unit tests are expensive to set up and maintain.
The question is not whether unit tests are beneficial but rather if these benefits are worth it.
Developers constantly need to think about how to design their tests in a meaningful way, how to mock stuff, how to get more code coverage, and whether they covered all potential edge cases.
On top of that, when parts of the code change, they might have to refactor dozens of tests, no matter how well they were designed in the first place.
And all of this can lead to engineers becoming very shy to touch existing code that has been cemented in place by excessive unit testing. They tend to avoid rethinking existing solutions, trying out new things, and being bold in general.
𦧠What can you do instead?
1. Use a strong toolset
You can catch a lot of stuff in advance with a strong toolset. Code reviews, linters, strict type checking, and some solid coding guidelines provide a good basis to produce a robust product.
2. Manual testing
If you do need to test something, test it manually by hand. Yes, you will repeat yourself, but you are probably closer to the production environment and can easily test scenarios that might not even be testable automatically.
3. Rely on users to find bugs instead of engineers guessing them
It's likely you are already addressing the obvious stuff. But most of the bugs you'll face come from edge cases that you couldn't have guessed in your wildest dreams. If you accept that code will inevitably break in production, your focus will shift more towards how to fix it fast.
4. Add safety nets
Speaking of fast fixes. Focus your effort on creating safety nets to detect and address bugs as quickly as possible. These include gradual rollouts of new features, using good bug and performance monitoring tools, and feature flags to quickly roll back in case anything fails.
5. Write stuff down
If you run into weird edge cases, leave a quick comment in the code to explain why stuff is in there and discuss it with your teammates. That way, you can remember everything when trying to refactor pieces of your code.
6. Code is written to be rewritten
Establish that in your team. You will never get your code right on the first try. Encourage frequent improvement and don't let anyone shy away from touching someone else's code.
7. Work on your code, not on the supporting structures
If your code is fragile and keeps breaking when you add or modify parts of it, focus your energy on making it more robust instead of making the tests around it more complex.
π€ So, should you ditch your tests?
There are cases where the cost of a single failure is simply unacceptable. Take banking, for example, or critical infrastructure. These systems need to deliver 100% of the time, and it's just not worth messing with that.
Also, at a certain scale of a company, unit tests will for sure be worth it. When there are too many people involved and things keep constantly breaking.
But this point comes later than you probably think. Netflix apparently went to 100M users without a robust testing suite in place (Ex Netflix engineer on unit testing).
So if you are a startup or medium-sized company and your priority is moving fast, then you can probably get away without unit testing for longer than you think. And going even further, it might even benefit your company because you can focus on setting up processes to fix bugs faster.
What do you think about unit testing?
EDIT:
Going without automated tests is not an excuse to go wild and crazy with your code. The time you gain needs to be invested into alternative failsafe mechanisms. It's like driving a sports car in race mode to make progress fast, but you better know what you are doing. It requires you to be always alert and you need to feel comfortable with running into some road bumps along the way.
Side notes:
If you want to dive deeper, check out this interesting discussion on the topic:
https://www.youtube.com/watch?v=pvBHyip4peo
Need a helping hand on your Vue/Nuxt project? Reach out to me through https://nuxt.wimadev.de (We do tests)
Top comments (64)
Interesting article with a provocative headline.
What sounds like a generalized rant against unit tests should rather be an inspiration to write better units tests, not ditch the supposedly dysfunctional existing ones.
Very interesting take, might be good in practice for edge cases - BUT a business or otherwise critical production system SHOULD be stable and tested BEFORE shipping to users!
This explains everything! But you should have quoted the whole motto which is "Move fast and break things" said by Mark Zuckerberg, founder of facebook and pulling the strings behind Meta.
If Zuckerberg is your role model, forget about everything. Ditch quality! Don't test your code! Use React, move fast, break things, and hope your customers will still use your products. But seriously, I'm sure even Facebook's software contains unit tests.
Consider investigating what's wrong about your existing unit tests and how they should have been written instead to boost quality AND productivity instead of making your developers shy away from refactoring.
Are you suggesting that Facebook is full of bugs? Or that it has some awful performance? Seems to have worked out pretty well for them as far as I can see.
Have you used React? Did you like it? Have you used Facebook? Did you like it?
But more importantly, do they really move fast and break things without testing? I don't think so. While they introduced breaking changes to React, I heard they don't update their own codebase accordingly.
But in the end, I can't tell. Luckily, I didn't have to work with React anymore recently.
Yes, to both and yes to both :)
For my purposes React is fine, it has foibles, what doesn't though. I think it really depends on what you are building and for what audience. My audience isn't going to notice a few more 'k in a download vs the power the product they've bought provides them. I'm sure I'd feel different if I was targeting a consumer audience with a short attention span - but the speed of releasing features and the abilities I have for separated development teams on a single product work for me.
I agree! Removing tests entirely is a fairly radical approach and certainly doesnβt work in all industries. But I like the idea of challenging these set processes that are often just taken for grantedβ¦ Lot's of companies (especially in Germany I feel like) are very risk aversive even if the cost of failure is comparably low
The big companies OP mentioned are "entertaining" softwares, it doesn't matter for a social media website has some bugs, or strange behaviour on Netflix interface.
For other more serious industries, like banking, retailing, medical, motor, etc., the cost of relying on users to find bugs is lawsuits π
Unit tests are a trade-off. What are the costs of making and maintaining unit tests? If there are no unit tests, what are the costs of end-users finding bugs and fixing them? What are the soft costs between unit tests being a forcing function to embrace SOLID, WET/DRY, KISS, YAGNI principles, versus code that is rife with high coupling, low cohesion, inflexible, and hidden dependencies?
For some software, it doesn't matter if the crash rate is 5%. It may not matter that a good portion of the developer resources are spent fixing an endless and growing portfolio of bugs. End users just restart, and continue.
For other software, if the software crashes and that results in people get hurt or expensive equipment is rendered inoperative, 5% may be unacceptably high.
Totally agree here! There is no right or wrong answer for every scenario, it really depends on the software and company. Writing tests is a bit like paying for insurance. If an incident can completely ruin you, you better invest upfront, but the decision to get it for your phone is debatable...
The trade-off is "move fast and break things" vs take your time and build something that lasts forever. Most companies choose speed. Most users would prefer slow.
Unit testing is a discipline. Theyβre a tool to help you keep track of the assumptions and expectations youβve assigned to the code youβre writing. The larger your code base gets, no matter how clean or simple it is, it eventually stops fitting in your head. Unit tests ensure that these expectations and assumptions are not lost.
Why not keep track of that in a spec sheet or through code comments? Much easier to write and maintain
Because whatβs easier for you today is a nightmare for the next person trying to make changes.
Unit tests are executable and can actually validate the truth of your code. A comment nor a spec sheet will ever do that for you in any reliable away.
Also if its really hard to create unit tests, that says more about your code than it doesabout the process of writing tests. Unit tests should be the easiest aspect of your code base to not only write, but maintain going forward.
Comments are the first thing to become neglected in a fast moving software project. If no one is actively working on it, you cannot trust it's validity. Additionally, the process of testing exposes bugs that you would have missed. It also results in code that is easier to extend and maintain.
Spec sheets always get out of date. Code comments just the same. There is no syncing back after you go add that little functional change. Which ends up being 80% of the codechanges during the lifetime of the project.
I've been a professional developer for almost 30 years, and have almost never used any automated testing, and it's never been an issue.
Yeah, that totally works if you do pure UI and or integration stuff
and skip heavy logic parts.
Unit tests do not work well for that use cases.
Unit tests are there to test logic and/or state transitions (like in a state machine).
I've done plenty of that kind of work too. Without automated testing (unit or otherwise)
Interesting. What kind of apps are you working on?
I've worked on all kinds: desktop apps, front and back-end web stuff in a number of industries (OTA, e-commerce, fulfillment, payment gateways, image and media libraries, property management), and most recently desktop game development (UI)
How much were your companies spending on manual testing?
No way of knowing, since it's part of the development process. How much money is spent on writing and maintaining unit tests?
What happened when you found a software bug? Who caught it? How long did it take to fix? How complex was your code base?
Pretty tricky questions to answer really as I've worked for companies from tiny start-ups with very few developers - to big companies with 100s of developers... and code bases ranging from simple small libraries to large, complex systems.
I've always found that not using automated tests encourages a more hands-on familiarity with the code you are working with, and a deeper understanding of what it actually does and how it does it... better equipping you to fix and diagnose issues faster. Automated tests have always felt like an extra layer of abstraction that merely serves to slow development and maintenance... whilst also encouraging the siloisation of project knowledge (never a good thing for projects IMO)
I work a lot with Nuxt (a framework atop Vue.js) lately. I barely find valid cases for classic unit tests here. Often you "only" fetch data from backend and display them. Or collect data from a form and send them to backend. This can't be unit tested, this has to be E2E tested. Okay, maybe I can unit test for example getters I write in my Pinia state stores, but they are usually so straightforward that you just see the results immideately during development and once estsblished, they are quite unlikely to change. So in this case having to write dozens of dumb unit tests that just
"test" whether you can use
Array.prototype.filtercorrectly is quite a waste of time, if you ask me.I recently came across the concept of "visual testing" with Backstop.js - it opens your app in emulated browser, performs defined scenario (visit route, click a button, etc), takes a screenshot and stores it for future reference. If you mess something up during development, the test starts failing, becuase actuall screenshot looks different. I think this is good way for quite complex and yet easy-to-manage (because you dont have to describe the outcome) testing. On the other hand there is some overhead as you have to store the screenshots somewhere.
Very interesting approach with that screenshot testing! Thank you for sharing!
No unit test != No automated test
I'm always more confident with integration tests and e2e tests than unit tests. Furthermore, they didn't break on refactoring. And it's always better to test actual user interaction instead of implementation details.
Manual tests are really expensive in the long term. Community tests are great but aren't applicable to any project.
That's true. There are layers to automated testing. And integration/e2e testing can give you more coverage across your system.
I can say though, that as soon as your app is just a little bit more complex and multiple people or teams are involved, e2e tests are really much harder to setup and they do require a lot of maintenance effort. You have to run and maintain a separate production like environment. And that causes tons of issues like multiple people trying to work on it at the same time, testing data being wrong or outdated, and thinking about all the underlying infrastructure stuff...
Indeed test cases need to be rethinked.
It's about trade-offs between simplicity, side effects, replayability and maintenance but if you recreate the full test situation and delete it after the test, you can deal with those issues.
This way, you should be able to launch your e2e test on every environnent you have credentials for (no need for production-like environment, any dev or review envs will match)
For integration test, no environment is required, with test containers, or embedded databases/queues and tools like wiremock you should control you own dummy environment.
Totally with you on that.
EDIT:
Going without automated tests is not an excuse to go wild and crazy with your code. The time you gain needs to be invested into alternative failsafe mechanisms. It's like driving a sports car in race mode to make progress fast, but you better know what you are doing. It requires you to be always alert and you need to feel comfortable with running into some road bumps along the way.
Certainly, there are coding practices which may significantly reduce number of unit tests. But by completely ditching unit tests we increase mental overhead and feeling of uncertainty for developers. This, in turn, basically kills productivity and it gets much worse than with unit tests included. Lack of unit tests also makes many refactorings much more complicated. Refactoring in majority of cases is more time effective way to improve code than rewriting from scratch.
Yeah, guess it probably won't work for every team. You have to gather some head-through-the-wall people to pull it off.
I can see where you're coming from to a point, but I think there's something fundamental missing - it's good code that has been designed to be modular, clean, and therefore very unit testable, that you can potentially require less unit testing of. Anything less than that is usually short-sighted and doesn't consider the potential lifespan of the code and how much time you can save in the long run.
Absolutely! Going with less tests is only possible if you invest that saved effort into your code. It doesn't mean, do whatever you want...
"2. Manual testing
If you do need to test something, test it manually by hand. Yes, you will repeat yourself, but you are probably closer to the production environment and can easily test scenarios that might not even be testable automatically."
This is what I have always been doing. I am new to automatic unit testing, have read about how to write it. But I am always worried about time to allocate for it, instead of focusing on improving the code itself to become robust, after manual testing ( as well as manual integration testing) and rethinking over the code in accordance with the goal of the app/project.
It really depends on your project. If the stakes are comparatively low and you have a quickly evolving codebase, I think you can go very far without the need for automated testing.
I think tests are essential for good software but maybe not a priority when starting an MVP project.
Just consider which tests to write, as some tests are not as valuable as others, and they make future change harder, because if the project keeps changing often then you need to go in and change all of the tests.
I believe that unit tests are written poorly the majority of the time -- they should test the functionality of the code, not just the happy path, but most of the time I see the failure path(s) ignored because the code coverage threshold had been met. This makes ensuring the signature of that unit less reliable and any refactor will most likely miss preserving it.
I find the idea of testing in production behind flags or certificates a better solution to the problem. You write the integration and end-to-end tests against that system with known dummy accounts, and those tests should run often enough to avoid breakages. Once a feature is passing the tests and has a level of completion to release the flag can be removed.
This could be something that a more senior developer could setup and maintain, while the business code can be done by the entire team. Having this in place would ensure that good test coverage that is aligned with the requirements.
Is AI already playing a role in facilitating and improving the process of unit testing?
Copilot already does make life easier. It's going to be very interesting what's coming in the near future
Good tests are documentation. When I am to change some code, I often go straight for the unit tests and look at the overview module in IntelliJ to read all the method names. In a good system, they will tell me most of what I want to know. That's super useful when the original coder is gone, the original spec is nowhere to be found (which is the case in 95% of the places I have been), no one on the team has any idea about that feature anymore, etc.
The best unit tests are the ones that actually exercise large parts of the system and application logic without doing I/O. Pete Heard of LogicRoom calls these "Goldilock tests". You get to test 90% of your application logic with almost the same confidence you have in integration test, but they run much faster, so you can afford to test many more cases. This requires a very clean architecture though, to allow it.
Bad tests are just wasted effort: testing of getters and setters, etc. Ugh. And people need to kill tests that are duplicated by other tests. Strive to trim the fat.
Very few practice TDD, but when doing so, you often end up with better architectures and docs to boot.
but but but everything under (insert arbitrary percentage number here) test coverage is risky!
Coverage is everything π
Some comments have been hidden by the post's author - find out more