Testing in production isn’t for everyone.
In fact, testing in production used to be considered a result of amateur programming. It was associated with shipping under-tested products that had poor UX and low retention rates.
This meant that in the development and QA world, testing in production was akin to cutting corners and delivering poor quality software.
In fact, this is no longer the case.
Though the phrase still strikes fear into the hearts of developers and engineers alike, the QA community has largely come to believe that testing in production has its place - when done right. Although, it is still the case that when done wrong, the implications can be massive.
But should you be testing in production? What are the benefits, and what are the drawbacks?
Keep reading to find out!
The case for testing in production
Testing in production improves product quality and actually helps build a more robust QA strategy. But why is this the case?
Let’s say you test in local staging. This is a typical testing environment, but it is not identical to the actual production environment.
The production environment may have different data sets, assets and configurations, meaning the user experience is going to be different from that of the testing environment. This runs the risk that your testers have investigated one user experience, and what your actual customer sees is something different. You don’t want your customer to be the person discovering bugs. Of course, that should be the testers.
Testing in production, therefore, has considerable value.
But don’t just trust us on the subject.
We decided to speak to three testing professionals who actively use testing in production, in order to gain insights into the benefits of the technique and the drawbacks.
Benefits of testing in production
So, what are the benefits? Well, it turns out that there are a number of them!
You see what the customer sees
There is no margin for error in terms of production environments. You are testing in the real-life environment of the user, so you can effectively test their user experience.
“You see what the customer sees. Your experience is the customer's experience”
- Sheri Byrne - Head of Accessibility for VMware
You can prepare for the worst
Testers are used to testing software. But customers - well they are almost certain to break something in your app. And it’s best to be prepared for when the inevitable happens.
“No matter how hard you try, you will never cover each and every use case in your testing. This is because your testers still understand the point of the software. They've probably been involved in every step of the design process. So, while they may try to break it, they'll never be as successful as someone who's never seen it before. The customers will always find the edgiest of edge cases and break everything.”
- Shayne Sherman, CEO of TechLoris
You can use it to verify payments
There are definitely some elements of testing that benefit from the production environment. Ajeet Dhaliwal highlighted the importance of testing in production when looking at payment functions. You cannot precisely match the payment process, so it can be beneficial to conduct the test in the production environment.
“There are some cases where staging or QA environments cannot precisely match the production environment. While all functional testing should take place in a non-production environment there are cases where you want to verify flows in production, for example, payments. In the crypto space for Blockchain, while there is Testnet, to validate correct implementation for users, some testing on the Bitcoin blockchain may be necessary.”
- Ajeet Dhaliwal, Lead Developer and Founder at Tesults
Drawbacks of testing in production
But testing in production does not always work. In fact, it can have some real drawbacks if it isn’t used correctly, or if it is used unnecessarily.
As explained by software engineer Cindy Sridharan in her extremely thorough article on the subject:
“When done poorly or haphazardly, “testing in production” does, in fact, very much live up to this reputation. Testing in production is by no means a substitute to pre-production testing nor is it, by any stretch, easy.”
What Cindy means is that testing in production is very difficult to get right. Assuming this is an easy fix would not be correct.
Here are the drawbacks our experts highlighted:
There are definitely restrictions
There are some big restrictions to testing in production. In fact, some QA experts and engineers hear the term and run a mile. That’s because there are a lot of scenarios where testing in production is a big ‘no’.
“You can't do load or performance testing because that would degrade everyone's experience, including your customers. You can't use customer data (HIPAA/GDPR/CCPA laws) so it is all fake data, for example, fake credit card #s that never go all the way through the paywall.”
- Sheri Byrne - Head of Accessibility for VMware
You risk customer churn
By testing your product quality in the production environment, there is a large amount of risk. Your customer may come across issues, churn, or simply write off your software as buggy. That’s why companies who test in production need to have the ability to take risks, and not suffer hugely if they go wrong.
“The major drawback to testing in production is that it's already there. You've probably gone through an extensive deploy process, maybe even brought your site down for a bit. Now it's there for all your customers to see. If you haven't tested it yet, rest assured they will find problems and let you know before you do.
The biggest problem isn't with your existing customers, though. If a potential customer happens across your site while it's in a broken state because of some bad code, the likelihood that they'll convert drops drastically.”
- Shayne Sherman, CEO of TechLoris
When done wrong, it can be a catastrophic
As Cindy Sridharan, testing in production is a cautious art. You have to have a highly skilled team willing to take risks. Otherwise, you could end up with a real mess you did not want on your hands.
“Polluting production databases with test data is generally a bad idea. If a new feature is being tested and there is a bug, catastrophic issues can be introduced into production systems. I’ve seen some integration tests running in testing environments that as a precondition delete all rows from a table and then start the tests. You don’t want anything like that anywhere near a production system.”
- Ajeet Dhaliwal, Lead Developer and Founder at Tesults
In conclusion
Testing in production is a tricky subject matter. While it has its benefits, there is a cautious approach that the QA community takes on the matter - tread carefully, they say!
But that doesn’t mean you should run a mile from the prospect of testing in production. In fact, it can be a very useful tool in your testing toolkit.
Take a risk assessment of how your software could deal with the potential drawbacks, and see how it levels up with regards to the benefits. Testing in production could be right for you. Finally, our co-founder, Ronald Cummings-John, recently wrote a fantastic article on the subject on his LinkedIn, inspired by his conversations with QA expert Sally Goble. Check it out now!
