It’s been well-documented over the years how costly manufacturing defects to production lines spanning across a range of different industries. According to the Cost of Quality (COQ) from the American Society of Quality (ASQ):
Many organizations will have true quality-related costs as high as 15 to 20 percent of sales revenue, some going as high as 40 percent of total operations. A general rule of thumb is that costs of poor quality in a thriving company will be about 10 to 15 percent of operations. Effective quality improvement programs can reduce this substantially, thus making a direct contribution to profits.
If we can, for a moment, recognise that the ASQ is not directly speaking about software in this instance, how much do you think software defects (bugs) in your organisation might cost? Have you ever taken the time to determine this?
Research from IBM suggests that the cost to fix a bug after the product has hit the market is four to five times more than one found during requirements gathering or production phase and that cost only increases from there. There are three main areas you should prioritise when finding bugs:
Bugs during requirements gathering
If we look at the requirements gathering stage as the first stage, the bugs we are looking for are likely to be functionality-related, rather than code-related issues. Since requirements gathering is not coding, we can’t use automation tools here to find potential issues. What we do have though, is relatively low amounts of invested time during this stage. Because the code has not been developed, there is a potential to find functionality issues which might arise early enough to ensure that certain features aren’t developed and man-hours not committed.
An important function for the product manager at this phase is working with customers and potential customers to understand functionality requirements. Conducting customer interviews and interviewing potential customers is one way that product managers can determine the kinds of features and functions to build.
Working with the development team to then gather the requirements of these features and functions will hopefully uncover potential functionality bugs. Including QA at this phase is an important step that many people skip. It is not unusual for QA in most organisations to be set as a goalkeeping sort of function that only skips into action at the end. Having QA involved during requirements gathering can unearth all kinds of issues that may not have been contemplated.
Bugs during production
For the more mature companies out there, QA is most likely introduced during production. Some of the code is being tested automatically with unit tests, critical flows are being run through automated testing suites and some level of functional testing happens with a few folks in-house on a few devices. In this very common scenario, people are generally content with their QA operations.
When bugs are found, the development team will work on resolving them as quickly as possible and everyone moves on. However, there are many, many costs which have accumulated up to this stage. Costs of the requirements gathering can be tacked on to production costs of engineers, marketing, design and more. To find a bug during this stage will come at a larger cost to the organisation simply because more has been invested now.
Additionally, there are costs associated with future development to the company although they are harder to quantify. Because many organisations no longer have a singular defined production and maintenance cycle (e.g. they are using Agile/DevOps), engineers and developers are almost always working on something new or improved. If the developers must go back to an older feature because a bug was found, there are costs associated with context switching.
Bugs after production/release
Bugs found after production or release could be catastrophic. At this stage of the release and development cycle, the finder of bugs could be anyone from your QA team to the founder’s brother or investor’s cousin. The point is, that bugs at this stage do immense harm for a multitude of reasons. Some bugs might seem quite arbitrary but other bugs, perhaps ones that impact your critical workflows can impact the company on many levels.
A bug perhaps in your sign-up flow could cause countless users to never register. This kind of bug may never even be identified if your testing strategy does not do enough to cover each potential device and OS combination. Furthermore, these bugs impact future growth of the company but are invisible in nature – unless you know people aren’t signing up because they let you know - but that isn’t going to happen.
40% [of users] say they're likely to kill an app the moment they encounter a problem
At the end of the day, each of the different parts of your development cycle will have a different cost associated with them for finding bugs that only really escalates as time goes on. The real way that you solve this, is by working to integrate your QA process as early as possible. Don’t ignore QA by making it the last thing you think about before releasing.