How much does functional testing cost your business right now?

Most of the time, engineers and QAs think about QA testing as an operational commodity with a standard outcome and price.

We’d argue that that’s not necessarily true – i.e. that it’s possible to leverage functional and UX testing for global growth outcomes via things like Test-Driven-Development (TDD). But even with that said, this is an important topic worth thinking about. Testing is always likely to be a component of your production line; how much does it cost? How can you get it done faster and cheaper? How can you save time and money and retain confidence?

Below, we’ve tried to spell out: “how much does QA cost your business?” We’ve identified all the factors we can think of to help answer the question, “how much is your testing costing you?” 

So if you’re looking to save yourself time and money, read on. Here’s our account of what testing can cost.

Talk to GAT about cost

The many time costs of testing

Doing the work – test execution

The first and most obvious category of time spent testing is the literal time a team member spends literally testing. Many businesses run on a rule of thumb of three devs to one QA, where the QA will spend 80% of their time on executing tests if they have a fully manual suite.

We saved Booking.com 70% of their QA team time by taking on international regression testing including payments and exploratory testing from their team. We’ll get into how to think about labour time in a little bit more detail below.

Doing the work – test automation

In more and more companies, the emphasis is on automating more tests. Since test automation is generally designed to save the time cost of manual testing, it can be easy to write off or ignore the time costs associated. But they can be significant, and they include:

1. Test QA planning and strategy: automated or manual still requires that you write a quality strategy and plan tests into a release programme.
2. Writing tests: where engineers own quality, we’ve seen devs talking about spending more than 50% of their time writing tests.
3. Maintaining tests: in our own polling on webinars, flaky tests were cited as the most common reason that automation-first test approaches reverted to manual testing.
4. Automation skills = higher salaries: We’ll explore this more below; but where test automation is involved, the employee is likely to command a higher market rate as an employee which offsets the savings that automation promises.

Test automation is great. But it’s also true that teams systematically overestimate what percent of their tests they will automate, which can lead to under-resource in testing. When a company bets that they’ll automate all or most of their tests, they under hire – and the remaining QA can become a manual testing bottleneck.

Wasted time; time you can save or minimize

Thinking about time spent automating tests and time spent executing tests is an optimistic way of estimating time volumes. For example, engineers identify that 32% of their time is spent on non-value added work. We’ll get into time you can save below.

1. Time lost to context switching: coding is long-form work which requires long-form concentration. The average person is interrupted 31.6 times per day and it takes 23 minutes to refocus after an interruption. If interruptions happen at even intervals throughout the day, the above maths suggests we’d never reach full concentration in an eight hour workday. The point to highlight is that it’s not just about how long you’re asking your engineers to do something for, but the number of instances.
2. Time spent organising and managing devices: you probably lowball the different # of devices and environments you should be testing on. Managing devices is a hassle: the storage, insurance, breadth, cables, and most businesses prefer to outsource this kind of test 
3. Misjudged automation tests: we’d argue that a bias towards test automation among engineers who believe that manual testing “should” be automated 
4. Tests which you should automate: conversely, you’re likely to under-automate your tests. Often, the demands to undertake manual tests (urgent but not important) 
5. Time lost in the communication layer between QAs and engineers. This would include false positives (did you know: GAT has the lowest false positive rate on bugs in the industry?) and it includes QAs not including reproducibility details, which can lead to an engineer spending time trying to reproduce a bug. To minimize this, you need clear guidelines for what constitutes a bug and you need to clearly give the reproducibility details.
6. Time lost to meetings and irrelevant work. Engineers spend up to a third of their time doing work they feel is pointless, and Paul Graham does a good job of explaining why engineers in particular “pay more” for meetings than QA Managers or generic commercial managers

Each of these are targeted by Global App Testing to as areas to refine. For example, context switching is often a function of false positives which we've reduced to the industry-lowest at 5%. We do device management with our distributed crowd so you never have to worry about the specific devices you need.

Let's talk about saving you time

Thinking about the price of time

There are lots of ways that wasted time can impact your business. Part of this is about release speed – more time and more bottlenecks means slower feature development, which is particularly important in B2B feature "arms races." But this is also about efficiency for the business. What does it cost for individuals to do the work of QA? We explore below.

Salary time; what they might look like

Everything above is salary time; time that it takes an individual employee to complete a task. But how much does that cost?

At the time of writing, Jobs site Inddeed.com estimates that an average QA manager in the US earns $59,840 in gross salary before non tax benefits are added. If you require that your software engineer does manual testing, they’re likely to earn over $105,000. 

But that’s for middle-of-the road technology companies. Most global app testing clients are from tier 1 software businesses. And according to Levels.fyi, the average total comp for an ICT3 engineer at Apple is £178,910 ($221,595) with a total comp of £1.25M ($1.55M) for distinguished engineers.

This graph implies

Misjudged automation tests: we’d argue that a bias towards test automation among engineers who believe that manual testing “should” be automated 

Tests which you should automate: conversely, you’re likely to under-automate your tests. Often, the demands to undertake manual tests (urgent but not important) 

1. A one minute interruption with a 20-minute refocusing window costs a business $30 for a below-average-pay Apple software engineer
2. A one-hour meeting with four engineers in the middle of the Microsoft pay scales would cost Microsoft $500
3. Asking an engineer to spend 20% of their time thinking about QA could cost the business $40K per year, significantly more than the same volume of testing would cost via Global App Testing

Beyond salary – the true cost of employees

1. Salary overheads. It’s generally assumed that people cost 1.7x their basic salary depending on what country you’re working in (plus a number of other factors). That includes the equipment, desk space, tax, benefits and perks, the cost of onboarding them.
2. Productivity ramp. A new hire will take 3 months to become fully productive and is likely to be less productive after they have resigned. That means you could be looking at a productivity ramp time of as much as 4 months per employee.
3. The cost of hiring. Recruiter fees and manager time both go into hiring an employee, which can cost a substantial amount.  
4. The cost of bad hires. A bad hire is likely to impact your culture, make other employees less productive, to demand manager time and engagement where it is not owed. We’ve heard it argued that the true cost of a bad hire is approximately 3 x their salary. Mediocre hires are normally difficult to get rid of
5. The cost of managers. Further to their own salary, employees need to be coordinated and managed, usually by a more expensive and more experienced figure. 
6. Time lost to evenings, weekends. Full-time QAs usually work out of a single time zone. Although the “cost” of not working more than 40h weeks is factored into every alternative (GAT testers don’t work 100 hour weeks, for example) there is a timetable cost about 

The alternative: the cost of bugs in production

Everybody stops testing in the end. That means that at some point you have to make the call that further marginal testing is no longer worth it. We would like to say that we're rational in when we stop testing, but test fatigue or lack of test resources is more likely to comprise a bigger part of the strategy than we'd like to admit. In particular, businesses are likely to skimp on device testing and real cash testing.

Commercial penalties of bugs in production

1. Poor reviews on the app store and beyond. How important are the reviews to your business? Irritating software experiences are among the top reasons for poor reviews, and your review score is likely to be disproportionately impacted by bad reviews.
2. In-flow user and revenue impact. The savings that Global App Testing identified on checkout testing earlier in this article – such as that one GAT customer improved their checkout completion by 12% – are only possible because of the negative impacts bugs may actively have on your checkout right nwo. 
3. Generally poorer NPS and word-of-mouth. In order to deliver excellent and cheap organic growth , you need a healthy WOM recommendation, which is usually encouraged by a great NPS. 
4. Decreased user trust. In a recent episode of Building Globally Podcast, Derek Boman talks about the role of design as more important in financial applications because the burden of trust required by consumers to share their financial details with you is very high. If you're asking your users to take a high-trust action, such as entering financial details or personal information, then 

Compliance risks associated with bugs in production

One of the biggest risk for some features is that your software becomes non-compliance. Below are a couple of the largest softwar e

1. GDPR and other data-related laws. Failure for software flows which are designed to enforce compliance with GDPR to execute could result in a fine. 
2. European Accessibility Law. Do you have confidence that your software is likely to comply with forth
3. Industry legislation. Whether your software operates in finance, healthcare, or any other regulated industry, most of the regulation you are likely to have to worry about is going to be concerned with your specific industry. 

Reputational damage

For obvious reasons it is not possible to exactly quantify how serious reputational damage might be from a bug. But often, very serious. For example, Parasoft argues that on the day a business announces a software glitch publicly, they lose $2.3 billion of shareholder value.

To illustrate this, there are lots of examples of software glitches. Data breaches are among the most common – 74% of large businesses reported cyber-attacks or data breaches in the last year according to twenty four IT. But there are also lots of public examples of functional failures .American Airlines’ booking time off limits did not work in 2017, leaving 15,000 passengers stranded over Christmas.  

In the UK of the most high-profile news scandals in the last year related to a bug. The bugs in the Horizon system by Fujitsu were brought to the attention of the UK public earlier this year with the Post Office scandal where an accounting bug put innocent individuals in the UK in jail for alleged accounting fraud. The cost to both the post office and Fujitsu has been incalculable. 

Engineers allocated to bugs in production

Finally, the cost of sloppy quality is that engineers will be asked to take time out of new feature development in order to fix bugs which have made it to production, which are famously "100x" more difficult and expensive to fix. 

What should I do next?

If you're curious about how Global App Testing could save you money or time, you should get in touch via the button below and we'd be happy to put together a personalised quote based on your requirements.

Talk to GAT