What are the alternatives to unit tests?

So what is your opinion about this? Did I do unit tests wrong?

Probably yes (sorry!)

It's a classic issue of writing tests that are too coupled to implementation detail. People then get frustrated at tests because they can no longer refactor without changing everything

So I wrote a bunch of unit tests for every class, ending up with about 200 tests after the first version was released. Trying to hit that famous 100% coverage.

I'm going to speak in terms of TDD; and that does not prescribe writing tests for every class/function/whatever. It prescribes it for every behaviour. So you may write a thing that does X, internally it may have a few collaborators; don't make the mistake of writing tests for implementation detail. These are still unit tests.

Ask yourself. If I were to refactor this code, would I have to change lots of tests? The very definition of refactoring is changing the code without changing behaviour. So in theory you should be able to refactor without changing tests.

I would suggest looking into Kent Beck's book on test driven development. It's an easy read and quite short. Or if you like Go and dont want to pay any money have a look at my book. This video covers some of the main issues you talked about and probably explains what i've typed a lot better infoq.com/presentations/tdd-original

Writing tests effectively takes a while to get proficient at, but the fastest way to get there is to study and retrospect the effect tests had on your codebase

One of the important things that unit tests will do is to get you focused on SOLID, most notably single responsibility. It reduces the temptation to write "Swiss army knife" functions or massive blocks of if..else or switch..case code. When you work in short blocks of testable code it makes debugging so much easier. Likewise, if you find tests becoming elaborate, maybe some refactoring is needed.

When you're working on a team, having the unit test gives other developers a guide as to how a particular function should work. If they come up with use cases you didn't anticipate, it provides an easy way for them to communicate it. When you're primarily working on the backend, it gives you something to demo in the sprint retrospective/demo.

When debugging issues unit tests make it easier to locate problem areas both in integration testing and in production. Without having this testing you can spin your wheels trying to find bugs.

Alternatives to unit tests? I've had to do these when working with legacy code where there were no tests originally written. Usually, these tests were in the form of one-off sandbox applications that would exercise a particular function or set of functions, trying to track down a bug. I've found this to be more inefficient than writing tests to begin with, particularly when trying to deal with critical production problems.

Thanks for posting your experiences. ❤️ I have similar history with unit tests.

Nowadays, I no longer bother to test everything. I do not believe there is enough ROI in doing so for most of our apps. I mainly test business logic. And when I say that, I mean only business logic. I practice Dependency Rejection, so business logic code is purely logic and is very easy to test. I will highlight the difference with a couple of images.

This kind of design is what you normally see exemplified in unit test demos with interfaces being injected into business code. This makes "business code" not only responsible for business calculations but also handling IO. Despite those things being represented as interfaces, the code will likely need to know specifics like which exceptions are thrown or other effect details which are unique to type of integration. So it has the appearance of decoupling while potentially being still quite coupled.

This kind of design also creates a lot of work in unit tests, since you have to create behavioral mocks of the injected components. The fact that you need a framework to dull the pain is a hint that it is not an optimal strategy.

Instead, I do this.

Here, the business logic code (inner circle) has no knowledge of other components outside of its purview... not even their interfaces. It only takes data in and returns other data. If IO is necessary to fetch the data, the logic does not care about it and is not responsible for it. Only once it is fetched do you run the logic code. It is also fair to give the logic code an object representing no data (e.g. Null Object Pattern or a Maybe). This is ridiculously easy to test since all you only have to pass in some data and check the the output matches what you expect.

For example, I might have some logic code like this:

let createItem existingItem data =
    match existingItem with
    | Some i -> Error ItemAlreadyExists
    | None -> validate data
    // validate: check required fields, invalid ranges, etc.

Then have a test like this:

    [<TestMethod>]
    member __.``Duplicate items cannot be created`` () =
        let existingItem = Some { ... }
        let data = { ... }
        let expected = Error ItemAlreadyExists
        let actual = Logic.createItem existingItem data
        eq expected actual

How do I handle IO? I have an outer piece of code (I call a use case handler, or just "handler") which is responsible for tying the logic to other integrations (database, API calls, etc.) needed for the use case. Sometimes logic steps are interleaved with IO, and so the logic has different function/methods for each step. The handler must check the logic response from the first step and perform appropriate IO before calling the next step.

This design draws a very fine line between which types of testing is appropriate for which parts. Unit testing (even property-based) is appropriate for business logic code. Integration testing is appropriate for the integration libraries used by the handler. End-to-end testing is appropriate for the handler code itself since it may deal with multiple integrations. But the main win, and the most important thing to the business is the business code -- that decisions are correct. And this is now the easiest piece to test. The other parts are no harder to test than they were before, but still not worth the ROI for us yet.

(I'm a unit-test-addict :) )

Did I do unit tests wrong?
In my opinion unit-tests are documentation, so if your product change, your unit-tests must be rewrited. If you had to rewrite to many tests for a little change so maybe you should make your tests more flexible, or use them only to test the "freezed part of your code" (utils functions and algorithms).

Is there an alternative?
In case of API (constantly evolving) some tools create test directly from spec (Swagger maybe).

Are integration tests enough?
It's difficult to test only a function with integration test, the scope is not the same. But testing "GET .../user/1" return the good object it could be ok. I higly recommand to use unit-test to deal with user-inputs (POST requests) because you can test the endpoint with a lot of bad entry (and check for security, malformed, bad type, ...)

Is TDD a placebo?
Personnaly it's a security I love to have :)

I'm bookmarking this to read later (so many good comments!) but I'll chime in with a QA perspective:

If you're working at a place with a formal QA step, test your implementation, not your requirements

I've noticed in my devs' specs they'll have tests for things like "this has been called once", "all parts of this if can be hit", yada yada yada, and then there will be things like "it returns all this info", "the info is sorted properly", "the info is in the right format", etc.

Then if you look at my tests, they're "it returns all this info", "the info is sorted properly", "the info is in the right format", etc... things a user would see and that are in the story's acceptance criteria for the feature. Where I am, QA automation (end-to-end with a hint of integration testing) is a part of the formal definition of done, so a feature isn't considered done until both of us have written the same thing just at two different levels.

I haven't written any unit tests for Web APIs yet but here's my take on TDD:

In my part, I don't recommend writing unit tests for every class. Only for classes that changes behavior based on various arguments and conditions.

Writing unit tests helps me in various ways:

1. It validates my understanding of the requirement. There's a tendency for us developers to jump right into coding without fully grasping the requirement. Writing unit tests forces us to think and ask questions even before the actual coding. Which eventually saves us more time than rewriting code from previous assumptions.

2. It helps me make design decisions. That is, if a class is hard to test, it may still be broken down into smaller testable classes. Therefore, enforcing SRP (Single Responsibility Principle)

3. Acts as harness after refactoring and bug fixing. Tests should still be green after code changes. It's a quality layer that signals me that I didn't break anything.

4. Like @JeffD said, also a documentation. I've written and deleted a lot of unit tests. Requirements may or may not change in the future. You don't know when or if it will but for this time that it's true, its better to write unit tests than to write none in anticipation that it will just be deleted or changed in the future.

Hopefully, these insights helped you.

If you haven't already, you should read Joel Spolsky's excelent article Five Worlds. To sum it up - great programmers sometimes come up with tips and best practices that make perfect sense in the area they work in, but are not very useful and maybe even harmful in other areas.

I believe unit testing is one of these best practices. When it comes to library development, for example, unit testing are great. In other areas their RoI is too low to be useful, and other kind of tests should be preferred.

In my opinion, automated tests should be an automation of manual tests. It usually easy to decide how to test something manually. For example:

• When you are developing a library function, you write a small main that prints it's output for some hard-coded inputs.
• When you are developing a new feature, you run the entire program and use that feature to see that it works.

These workflows are intuitive:

• You can't test the new feature with a custom main. I mean you can - but that would be a lot of work to re-create the environment needed to test that feature, and in the end it won't be very effective because that temporary environment may be different than what you use in the actual program.
• You can't test that library function by running the entire program. I mean you can - but you won't be able to give it the inputs you want to test and you won't be able to directly see the output. Unless you use a debugger, that is. Also, having to run everything up to the point that function is used will make your cycles needlessly long.

Since the manual testing strategy is so clear, the automated testing strategy should mimic it. Use unit tests for the library function and integration tests for the feature. Some people will insist on unit tests for the feature, but that has the exact same drawbacks of manually testing it with a custom main.

Unit tests have the greatest ROI when either

• the feature is business-critical and/or is very unlikely to change requirements (to guard against regressions)
• the feature is an algorithm with plenty of edge-cases (to fascilitate develepment and guard against regressions)

On the other hand, unit tests have very low ROI when a feature is not business-critical and has requirements that change very frequently.

Note that the value of unit tests is like everything else: it depends.

As to alternatives, I’ve had cases where API tests (on a running test instance of the application) provided an immensely high ROI. Integration tests, in the sense of testing the collaboration of a chuck of your codebase, those have for me always had a low ROI, because of the effort in setting up while still only resembling actual production behaviour (due to the mocked parts).

Did I do unit tests wrong?
I can't say for sure, but what I can say if "Trying to hit that famous 100% coverage" is a nothing but a wild goose chase. To find out why, see this article: dev.to/conectionist/why-code-cover...

Is there an alternative?
Code changes happen all the time and unit tests have to change with them.
It's unpleasant but necessary.
However, if a large part of your architecture has to change (and this happens quickly/frequently) then the problem is not with your unit tests.
It's with the architects and the faulty/rushed decisions they make when deciding upon an unstable/unreliable architecture.

Are integration tests (black- or grey-box) enough when automated?
NO!
Unit tests and integration tests serve different purposes. They are complementary. They are not meant to be a substitute for one another.

Unit tests are meant to test code. They are like a defense mechanism against yourself (or, more specifically, against accidental mistakes you might make).
The idea is the following:

• you write a piece of code
• you decide what your expectations you have after that code has run
• you write some unit tests that make sure those expectations remain the same after you've made some changes in that area

Because it's possible that changes you make in some places, have undesired effects in other places. That's where unit tests come in. They tell you "No, no! If you continue with these changes, you will break something that was working well. Back to the drawing board!"

Integration tests on the other hand test functionality. They check if everything works ok when it's all put together.

Is TDD a placebo?
Certainly not. But like all things, it works only if used properly.

As a side note, don't be discouraged if your unit tests didn't catch any major bugs. That's very good! That means your a good programmer who writes very good code.
If your unit tests failed every time you ran them, it would mean you're very careless (or in love and with your head somewhere else :)) )

Think of it this way:
If you hire a security guard and you have no break-ins, are you upset that you have no break-ins?
You're probably feel that you're paying the security guard for nothing.
But trust me, if he wasn't there, you'd have more break-ins that you'd like.

From my experience unit tests are incredibly useful when developing code that is perfectly suited for unit tests, generally things that have predictable inputs and outputs, and don't have dependencies or global effects. On the other hand if you're testing boilerplate code with a lot of complex dependencies (i.e. an MVC controller) it's probably better to cover it with integration or acceptance tests.

You should move as much code as reasonably possible into unit-testable blocks, but going out of your way for 100% unit-test coverage leads to tests that aren't worth writing and updating.

Then there are tricks, like mocking outside services (so that you don't have to actually hit remote services when running acceptance tests) and comparision testing, i.e. not testing the contents of an XML document but just storing it and comparing output directly to it. When testing APIs I also automatically test inputs and outputs on endpoints against a specification, which is a pretty good way of testing both the endpoints and the specification.

I also think that unit tests are a great way to force yourself to write easily testable code, which is usually better structured than non-testable code :)

But in general code needs to be tested if you care about it working. Any endpoint you don't test will eventually be broken.

My take on unit tests is to avoid them. Write your software ina way it could be tested easily, as this will keep your code decoupled, will force you to explicitly inject external stuff and more.
But if you have a decent type system and a bunch of integration/end-to-end tests, unit tests are not worth the hazzle.
After all you dont care about implementation details as long as your module/component/insert-similar-here does the correct thing

When I was at uni, unit testing didn't exist. We had "black box" and "white box" testing (which kind of map to integration and unit testing). But if anything, the idea that developers needed to write any testing code at all was seen as a general failure of Computer Science. There was an emphasis on things like formal verification (so, using mathematics to verify that code is correct) and the hope that you could just specify what you wanted and the program would be automatically created.

So I'm not surprised that you lecturer wasn't a fan of unit testing. In some respects unit testing is an industry-wide wrong turn, but then unit testing is a lot easier than some of the alternatives (have a look at Z Notation - when I was at uni the course in it brought people to tears).

I think it totally depends on the expectations of the system, how much experience you have, and the risk you can afford.

One of the systems I've been working on for about 14 years is a CRM. It's probably about a million and a half lines of code with a few hundred movable UI components. At one point we had around 10,000 unit+integration tests but have removed many of them. The issue is it tolerates a fair amount of mistakes in edge cases as they typically don't impact many employees at a time because someone with knowledge of the business tested the feature before it went into production. My goal is to try to provide the business with a high ROI for development time and over the years I've seen what works and what doesn't. These days I typically use very few unit tests. In fact given the application I try to limit situations where I feel they are necessary at all causing fewer errors, faster turn around etc.

Most of my development is a UI, maybe some business logic, a model, and a DB. If I have tests I make a class with dependency injection just to test the business logic, and many situations there is no business logic to test. If there are very few critical paths or few people are using it I may also not add unit tests.

Financial portions of an application typically receive many more tests.

If you are a new developer unit tests may be useful to help you understand the potential issues with the patterns you use.

Unit tests don't replace integration tests. Or making tests to reproduce bug reports (seems you missed it the first time).

I think most new developers make things complicated enough to require tests because they are bored or because they think it is clever or they just don't think long term. It may unintentionally function as a training exercise. In my experience systems with a few repeated patterns over and over seem to stand the test of time much better. Much of the "complicated" code just comes from tried and true libraries you shouldn't be editing. The best business code is something someone else who isn't even a great programmer can sit down at and understand quickly so they can add additional features that are valuable to the end user.

You may however be in a very different situation. If you are writing a library to release, your company screams at you for every little error, your software will be installed in hardware and sold, it is customer facing, it is life or death, etc then I would change the way I write and test it accordingly.

As a note one way I reduced complexity (and unit tests) was to not be afraid to move complicated things to the administrative user space where possible. This also allows the business to build groups of people who may not be "programmers" but who can set up complicated business logic on a test server, test it, and move it to production without a developer even being involved. Your software will be more resilient to change. Along those lines I recommend moving anything that looks like a report to it's own department or at least its own thought process/repo.

/rant (since they canceled our fireworks due to rain) lol.

I'm going to shamelessly plug my own Intro to Property-Based Testing ;-)

But seriously, PBT is a good secondary layer to proper unit tests. I'll be glad to answer any questions!

To me, you should treat tests like features and features like wizards. As Gandalf has said, "A wizard is never late, nor is he early, he arrives precisely when he means to." To me, 100% test is always too early. By writing tests to 100% completion, you are saying that your features are 100% done, and that your product, in turn, is 100% complete. So, when that requirement came in, there was no room for it, thus forcing a rewrite of the system to accommodate it. Tests aren't just an assertion that everything is complete, but a measure of how much work the product needs.

Another way to think about it, is using the same quote, but focus on the last bit. "...he arrives precisely when he means to." Rather than testing what you say (code), you test what you mean (intent/behavior/requirement). Sometimes, we developers only know code, we don't know the requirements. If that happens, then any tests that we create may be worthless, as they do not express what was intended. Due to lack of communication, you did not anticipate a new feature, thus creating new work. Some may argue that dependency injection would have solved this, but unless you apply that to littlest model, there will be some way that this will get you. This is why agile was about building smaller and communicating faster.

I like to think about tests in a different context than TDD. Rather than testing to mean asserting, I like testing to mean trying it on, sort of like shoes. If I like the result, I will lock it in. This idea harkens back to when we started programming. Code a bit, complain about a missing semicolon, compile it, play with it to see if it works, repeat. With this same idea, we just gain two things: it is automated and no CLI required to input. This will bring a different mindset into testing. Rather than building the test first or last, it is with the feature. Code a bit of the implementation. Code a bit of the test. Code a bit more of the implementation, code a bit more of the test. Rather than the test being something to assert against, it becomes an explanation of your intent. This is what it means for a test to become your documentation. Of course, that should not be your only form of documentation. Just because it passed the unit test, does not mean it is correct behavior.