End-to-end tests are usually run on a testing or staging environment, and for good reasons. But there are instances when running your tests on production is not that crazy after all.
Why not on production?
There are good reasons for not running your tests on a production environment.
First of all, when your code is on production it's usually already too late for testing 😓. You want to run your tests so that your customers don't get a broken product, so usually you should test your code before it arrives into your customers' hands.
Secondly, you don't want to mess with production without first messing with staging. This is really a consequence of the point made above. If you haven't run your tests on a staging environment first and your tests break things up, they'll break production and that's - to put it mildly - undesirable. Tests are meant to find bugs, and bugs are unpredictable, you can never be sure of their extent. You may end up tearing down your production environment or even destroying data, who knows. On the positive side, it's better that your tests find the problem instead of your customers, at least you get notified 😝.
Another reason for not running your tests on production is the data noise. Imagine running you tests multiple times a day for years. You will end up having a lot of noise in your production data, which will make it difficult to extract meaningful information from it. And data nowadays is a precious resource. On a testing/staging environment you can just refresh the database every so often, if you ever need to, to get rid of all the data created by tests. Clearly, this is not so simple on production.
So why running on production after all?
Even if running your tests on production may not seem a good idea at first, it has some important advantages. Of course, as we will see later, you shouldn't run your tests on production without having a proper plan in place to mitigate the negative aspects we've just talked about. That said, let's see why instead you should run some tests on production.
Well, first of all, your staging environment is not what your customers use. A staging environment should replicate as closely as possible the production setup, however, your staging environment will never be exactly the same as your production environment. At the very least you will have different URLs, but you can also have different API keys for external services or variables that enable/disable things when on production rather than on staging. These are your dark spots in your QA strategy, and can only be removed by testing your production environment as well.
Also, the good health of your application doesn't come only from its code, but from a combination of:
- The integrity of the data it uses.
- The status of any external system (e.g. databases, storage services, etc.) it depends on.
- The status of the runtime environment on which it runs (e.g. the server, the PaaS, etc.).
All the above factors are necessarily different between your staging and your production environments. If your data becomes corrupt on production because of a bug you missed, your application may become unstable. If your storage provider has a hiccup, your user won't be able to upload their files. If you have wrongly setup some file permissions on your production server only, you may not be able to run some parts of your application or access some of the files it needs.
After all, end-to-end testing it's also a type of integration testing so you will be able to catch all the above issues - hopefully before your customers will notice them - with a properly designed test plan running on production.
How to properly run your tests on production
So we have assessed that running some end-to-end tests on production is indeed beneficial, but also that there are some caveats we should take into account. So, below is a list of good practices to consider when running some end-to-end tests on production.
Don't rely just on "production" for your tests
This is probably the most important aspect. You should at least run the same set of tests on your application before it reaches production, on a staging or testing environment. This will make sure that your tests won't break anything on production, but it will also catch any major bug before your customers do.
In an ideal setup, we would have a three-tier testing plan:
- Run a full regression testing suite on staging before the code goes to production.
- Run a smaller set of tests (smoke test) on staging after each commit and deploy
- Run an even smaller set of tests on production at regular time intervals
Test less, test often
What we really want to do here is to monitor the production environment rather than just testing it. This means that there should be a very limited number of highly meaningful tests, running continuously at short time intervals. This will ensure that your application is actually running and its major components work as they should.
The interval at which the tests should run may vary and it depends on how long the tests take to run. But a good value falls between 15-30 minutes.
Cleanup after your tests
Your production data is important. You can use it to improve your product, your customer experience and nowadays even to build powerful Machine Learning models. So it's important not to have unnecessary noise created by our tests.
A solution is to remove all the data that we have created during our tests. There are various strategies that we can use - and we'll have a separate blog post to talk about them in more detail - but probably the easiest approach is to use the same test automation to perform the cleanup.
Let's call it frontend monitoring
The practice of regularly checking that a website or web application is running as it should is also called monitoring.
At its most low-level implementation we have uptime monitoring, which simply makes sure that our services are up and running. This is usually done by checking the response to a call to a HTTP endpoint. There are many providers out there to chose from.
If we decide to run our end-to-end tests on production at regular intervals, we are essentially implementing some frontend monitoring (some call it user flow monitoring).
The scope and purpose of the two techniques is different but it overlaps in many points. Both are intended to make sure that a production application is running as it should. Uptime monitoring usually runs very basic checks that require instants to run, and that's why the monitoring interval is usually very short, down to 1 minute in many instances. The checks (or tests) performed for frontend monitoring on the other hand require longer to run, but are also more exhaustive as they will take into account also the User Interface subsystem and its integration with the backend.
Conclusions
The words test and production rarely appear on the same sentence and that's due to very good reasons. However we have seen that with a proper testing strategy, we can safely implement a frontend monitoring solution on production.
If you are interested in setting up a frontend monitoring solution, Frontend Robot has a test scheduler specifically designed for this use case.
Top comments (0)