Do you need a full test suite for new projects?

#testing #code

Blog post was edited to reflect thoughts I had that were not coming across clear enough.

We write code, we test code, we profit.

Code without tests is bad practice....
Code without tests cannot scale....
Code without tests is ugly....

If you were to ask me a year ago I would agree with these statements.

I have worked on some large projects in the past and code without tests in these environments can REALLY cost thousands if not millions to the organisation in production bugs. So I have developed a natural thought over time...

"We have to have a full test suite for our projects. Unit, integration and e2e tests.".

But recently I started to question myself.

For new projets, do we need a full test suite? What is the real cost of tests? Why do we test and can we get this from somewhere else?

In this short post I will share with you my thoughts on testing in new projects, and why I think we may not need as much testing as you think....

Why do we test?

Tests give us the ability to refactor, scale our code and give us confidence (just to name a few).

It's simple really.

But different tests come at different costs.

Unit tests tend to be the cheaper kind of tests to write, whilst end to end tests are more expensive. And most of the time your end to end tests become quite flakey and you end up spending or hiring people to manage this.

So why should we have a full suite of tests? Can we get this confidence from other places? Do we need all the tests layers at the start of the project?

The answer might be quite simple....

I think it depends on the project and environment you are working in.

Should you write a full suite of tests?

Should you have a FULL suite of tests for a new project? I think it really depends on the size of your project, customer base and how fast you want to initially move and what risks you are willing to take.

Let's take two projects for example:

Project A - Large organisation and large customer base, bugs can cost hundreds of thousands
Project B - Small start up and small customer base, bugs only really cost a bad user experience.

Now if you were to ask me last year, I would say that both projects regardless of scale need a full test suite. Unit, integration and end to end tests.

On Project A the cost of production bugs or issues to customers is huge. I have seen bugs cost thousands if not hundreds of thousands in the past. So we naturally add layers of testing to get round it and give us confidence, we even hire people to manage this level of testing too. It makes sense. Give this environment, scale and project we gain confidence through our testing suite.

Project B on the other hand, is completely different. Newly created startup trying to survive in the market. Do you need the full suite of tests here? Well yes you can, but the question is do you need to yet?.

Do we need a full test suite?

I think it really depends on the project and size.

A project used by millions of people vs a startup project is completely different, so the way you test them should be different too.

Hopefully a large project (millions of users) has had the time and resource to build up its test suite over time to help the organistion grow and give people confidence to scale with the code.

But smaller projects, companies still finding their feet in the market, what kind of test suite do they require?

For example, rather than writing a full suite of tests up front in a startup, why can't we test our platform manually? I know this is CRAZY right?.... Manual testing!? Have I gone back 10 years? Well no..... think about it.

If your application is small a quick manual test will give you confidence things still work. Or do you write some core basic e2e tests (cheap cypress tests) that test the important parts of the system? Can your users still subscribe to your service? Whilst your unit tests still give you a level of confidence down stack.

Test suites need to naturally grow over time with our projects and products and need loving and caring, but I don't think its worth investing all that time up front. Get the core basics right and grow your test suite over time as your requirments change and business grows.

Just to summarise I'm not saying testing is bad.

I'm questioning the fact do we need a full suite all the time? Do we need to default into the mind set of having unit/integration and e2e tests for all projects?

So the question I ask myself:

Do you need a full suite of tests when starting a project? The real answer is probably not.

Wrapping it all up

Do we need tests? Yes.

Do we always need a full suite of tests right now on this new project? The answer is probably not.

When kicking off a new project it might be worth thinking:

"What testing do we need? Are there other ways we can gain **confidence* in our product"?

Confidence can come from other places. In early stages we can manual test, we can monitor in production and react and fix. We can contract test or move our level of testing down stack into units if we can.

We need to make sure we write tests when it makes sense and maximise our value from tests. We need to grow our test suite with our business and requirement and remember we might not need them all up front.

For future projects I will definitely be thinking about the level of testing required, which will depend on the environment, customer base and cost of bugs.

I wondered what your thoughts were?