The buddy.works webinar; and an algorithm
Over the last few months, we've been thinking hard about automated testing and the place of manual testing alongside it. In the last few days, buddyworks launched an amazing webinar featuring our senior developer Karol Toplolski where he set out one algorithm we've come to to help businesses identify what they should test manually and what they should automate.
You can view the buddyworks webinar here.
Why do businesses believe they will automate so many tests?
In TestRail’s first annual survey in 2018, businesses set out their plans for test automation. The 6,000 respondents automated 42% of their tests and planned to automate a further 21% next year.
But they didn’t. In the 2019 survey, the same 42% of tests were automated, and this time, businesses said they would automate 61% in 2020. By the most recent survey in 2021, just 38% of tests were automated. By now, the pattern is consistent. Businesses systematically overestimate the amount they will automate.
But why?
Why do businesses want to automate tests?
Teams tend to like the idea of automating tests. That’s because:
- You can run automated tests whenever you like
- Automated tests return instantly
- Automation is theoretically a one-time investment, making it cheaper to automate over the long term
And then together these factors lead to even better second-order effects:
- You can remove bottleneck slowing down your releases if your tests are instant
- You can improve your DORA metrics as you measure your progress
But the reality of testing difficulty belies this. We ran a survey during a separate webinar about the top reasons businesses don’t automate more tests. The top result (28%) of respondents cited flaky tests. The second result (26%) is not enough time.
Typically, flaky tests (28%) are a result of automated tests written for a volatile part of the product. They break as the product changes or iterates, meaning that automated test pools “decay” when built prematurely. A decaying or flaky test set requires devs to remember which tests they think they trust, meaning that information your business relies on to work is not in centralised systems but in the heads of individuals who may leave or forget. Having a trustworthy pool of automated tests guarantees the quality of your product, which is why we recommend a few rounds of manual testing before automating a test for a workflow. We can think of flaky tests as the time cost of maintaining tests.
“Not enough time” (26%) is more straightforward reason that businesses don’t automate more. But it’s also interesting to look at the vicious cycles that can affect time: poorly written code, or failure to estimate testing workload, can delay automation targets, which in turn create more work. We think of this as the time cost of building tests.
What’s the equation to calculate whether a manual or automated test is better?
ST + (ET x N) = the true time cost of testing.
You can check this for automated and manual tests to identify whether it’s cheaper for your business to execute a test manually or to automate it.
ET is the execution time. We know that automation is much faster here, and it’s the main metric businesses focus on when they want to automate all their tests. For Global App Testing, we offer 4-6 hour test turnaround with real time results. Tests land in tester inboxes straight away, so in many cases the first results come through much faster.
ST is the setup time including any maintenance time investment. It takes more time to automate a test script than it does to quickly test something or to send it to a crowdtester like Global App Testing. Setup time is also the second barrier to setting up tests, so it’s worth running this algorithm twice – one to add up which is more expensive, and one with adapted algebra to calculate the maximum time your business can invest in one go.
N is the number of times a test will be used before it flakes. It’s great that execution on an automated test is very rapid; but the saving is immense on a test used 1000s of times. If the test will be used twice before it flakes, the return is less impressive.
A final note is to ensure you know what you’re optimizing for. Is time or money more important? The labour costs of the individuals setting up the automated test (developers) versus the labour costs of individuals executing tests (global QA professionals) could be different; and try running this algorithm with both units plugged in.
Where can I watch the webinar?
You can watch it below.
