The Ultimate Guide to Regression Testing

What is regression testing? What regression testing tools are available? How do you perform regression testing? We're here to answer all your burning questions in our ultimate guide.

 

 

627404_FeaturedBlogImage_2_111519

In short, software is eating the world. - Marc Andreessen

It seems quite ludicrous to think that this now infamous phrase was written almost a decade ago when the world was even less consumed by software.

Almost every company is a software company today in 2020. And, if indeed you are a software company, then undoubtedly you need to be sure that your software is of high quality. One of the primary ways that you do this is through thorough testing.  

The thing is, consumers today are a fickle bunch.

According to research from Helpshift, 80% of apps are deleted after one use! This means that you’ve got a very small window to impress your users.

Provide anything less than incredibly high-quality experiences and you are simply tempting fate. It can be easy for a single bug to destroy that user experience that you’ve worked so hard to build. With unforgiving users and software development costs running anywhere from $50,000 to $250,000+ per project or release, it becomes incredibly clear that investment in proper testing is of paramount importance.

Part of your software development strategy needs to be focussed on both regression testing and functional testing. 

Your developers should be executing unit tests (more on that below) to ensure that their code is of high quality, that goes without saying. However, looking for perfect code to prevent bugs is a fools errand.

Set aside the right time, budget and resource for testing and you’re going to be further ahead than your competition. 

This guide is designed to help provide a fundamental understanding of regression testing for companies of any size and shape. It doesn’t matter if you provide software for consumers like Amazon or complex business software such as Salesforce. Regression testing forms an essential component of any good testing strategy.

So, let’s start off by diving deep into defining exactly what regression testing is… and what it isn’t!

What is regression testing

 

 

Regression testing is defined as the process of re-running functional or non-functional tests to make sure that the software hasn’t broken in any way after new code has been deployed. Simple enough?

Think of regression testing as testing your food as you cook it and adding ingredients according to a recipe. 

As you cook your food, the dish will change based on a whole host of different factors. If you add seasonings then the flavour of that dish will shift. This would be a regression.

The same holds true for software.

In the early days of social media platforms, the concepts surrounding hashtags hadn’t yet been invented. So, if you were a platform deploying hashtags for the first time, you’d need to verify that the release which included this feature hadn’t broken other parts of the platform for users.

To do so, you’d perform regression tests.

What is a regression test used for?

Regression tests are used in all manner of software development and they are used to keep the user experience working as it should after a new feature is developed.

Think about some of the devices you have in your possession. 

Most of you reading this will have some kind of mobile device and that mobile device likely gets relatively frequent software updates. Imagine if a routine software update blocked you from accessing the internet! This is why we use regression tests before releasing new code.

Unfortunately, if you’re the owner of a device, the experience of a device becoming corrupted by a bad software update will likely turn you off from ever getting one again. 

Yet this is exactly what happened last year with iOS 13.2 and Apple’s HomePod device. Apple quickly removed a software update that was found to be “bricking” certain models after users complained vocally. 

However, it just goes to show that even some of the biggest software companies in the world don’t always get regression testing right.

Regression testing isn’t just for hardware though. The same applies for just about any piece of software, website or mobile app you use. 

Proper regression testing eliminates the likelihood existing functionality will fail.

What is the difference between retesting and regression testing?

Retesting is a term we often hear people confusing with regression testing.

The two terms, while similar in nature, couldn’t be more separate in the world of testing.

An example of retesting might be, if we bring it back to the cooking food example, the process of checking the temperature of the dish. Nothing has fundamentally changed with the recipe or cooking procedure at this stage so we aren’t performing any kind of regression.

On the other hand, if we’ve now added in some new seasonings, a regression test will let us know if the taste of the dish changed or not. 

We can equate this to testing in a production environment a few months after a release has been completed. If no changes have happened to the codebase we might choose to rerun some tests that we have already run.

Hence the term, retesting! 

Automated regression testing

The thought that enters most peoples minds almost immediately is the concept of automated regression testing. After all, most regression testing is repetitive and manual so why not automate it? 

Regression tests, in a sense, are the perfect automation candidate in certain circumstances.

Automated regression testing takes the concept of regression testing and finds ways to reliably, and cheaply, perform tests without much human interaction.

The history of automated testing goes back of course much further than just regression, however. 

Keizo Tatsumi put together an incredible timeline which dates back to the earliest days of computer programming - 1962!

test automation

Courtesy: Keizo Tatsumi

Automated approaches to regression testing make sense when developers are able to appropriately write and maintain the scenarios required to complete the testing. This means that developers are responsible for maintaining proper test scripts.

Over the years, the role of test automation engineer has become more popular alongside tools for various web applications. Ultimately, your automated regression testing will only be as good as your regression test scripts.

Regression test scripts

Regardless of how you perform regression testing (manually or automate), you’re going to need a robust set of regression test scripts.

Regression test scripts are the foundation of your regression.

Unfortunately, regression test scripts are quite likely the most time-consuming part of any QA professionals job! Due to their critical nature, you have to be writing scripts that cover every possible scenario you can think of. 

Here’s where the problem lies; if you need to cover every possible scenario, what happens if you don’t?

Regression test scripts only test what you tell them to - automated or not.

The person that writes the test script has to make sure that it takes into account all variations and flows which is what creates a massive amount of work and upkeep.

Any time that something changes in your product or software, those regression test scripts have to be checked for accuracy. If you don’t check them and they fail, you won’t know if the failure is because of a bug in your code or a poorly written test script.

Automated regression testing tools for web applications

One of the more interesting developments in recent years has been the proliferation of testing tools designed for web applications to perform automated regression testing.

Automated regression testing tools for web applications are designed to take as much of the effort away from humans and spread this load to machines. Sounds great right?

In theory, automated regression testing tools are fantastic because they free up your team to focus on more important functions. However, the answer is much more complex and nuanced.

As discussed above, any type of regression testing is only as good as the test scripts used to execute them.

Which leaves us in a predicament that applies to any kind of tool you use for regression - it’s only as good as the input.

At the end of the day, automated regression testing tools for web applications can save significant time, energy and resource as long as you are prepared to put in the effort to maintain them properly.

Newer tools in the near-term future will solve these kinds of issues by building “smart” test scripts that require much less handholding! 

Regression testing in agile

The age of agile is upon us!

Agile testing leaves very little time for documentation. It relies on quick and innovative test case design rather than elaborate test case documents with detailed steps or results

- Nishi Grover

The above quote might seem like a little bit of contradiction to what we’ve been talking about so far with regression testing needing to be so very specific in order for it to be effective.

Regression testing in agile simply requires that the agility of software development cycles and sprints are taken into account. Modern engineering teams will already be agile and most like pursuing some level of DevOps too.

This means that in order to regression test in agile, the right systems, processes and procedures need to be put into place.

You’re still going to need to build out your test scripts and decide on the level of coverage you want to achieve.

This where a proper regression testing strategy for agile becomes vitally important.

Regression testing strategy in agile

If you are developing in agile (and if you aren’t, why not?) then developing a cohesive testing strategy that encompasses regression testing should be at the top of the list.

The following diagram from the book, Leading Quality, illustrates how testing shifts from investigating to verifying and this is where you need to build in the right level of regression testing.

investigating

Source: Dan Ashby

The level of sophistication of your company and team will inform your regression testing in agile strategy over the long term. There are three basic phases of all products:

  • Validation: product-market fit
  • Predictability: creating a stable infrastructure for scale
  • Scaling: minimizing negative impact to unlock growth

Your regression testing strategy should adjust based on the level of maturity in your product. 

Don’t simply assume that you should start automating every part of your regression testing because you will spend valuable time that could be dedicated to the product development itself!

When things begin to break, it’s not necessarily a sign that your QA has begun to fail. It could just indicate that your needs have changed and so you need to change your strategy to succeed.Excerpt From: Ronald Cummings-John and Owais Peer. “Leading Quality.” 

Regression testing in a sprint

One of the many benefits of regression testing in a sprint is that it can happen as soon as the code is put into the deployment flow. Depending on your setup, this means that regression testing in a sprint can be called by any number of methods such as your CI/CD tool.

Why does it matter that you perform regression testing in a sprint anyway? 

Well, according to Dan James and Bryan Aho it could save you a lot of money - upwards of $5,000!

Who doesn’t want to save $5,000? 

Ultimately the ability to save time and cash is a major boost for any development team!

Testing often gets the least amount of time dedicated to it during a sprint so maximising your efforts should be top priority.

Regression testing vs unit testing

People who are relatively new to the world of QA often ask what the differences are between regression testing and unit testing. It’s an easy mistake to make but there are some important reasons why the two are simply not even in the same world.

You can think of unit testing as the individual testing of specific components in your software or product. It is the smallest amount of testing that can be done.

Compare the unit test then to a regression test and you can see that regression is the sum of all the parts. 

Unit tests will always need to be done by the developer and should be built into the code itself. 

Regression tests can be created by anyone and ran externally from the codebase. 

Regression testing vs functional testing

A big debate in the world of software development QA is the role of regression testing vs functional testing. 

Can you get away with simply running regression tests alone?

We like to think the answer is no. 

While there are plenty of companies that rely on regression testing alone, functional testing is the only thing that provides a completely holistic test of how your app actually works for the user.

Sure, regression testing will tell if you a certain feature no longer works like your checkout flow but it won’t tell you if the new feature you’ve built actually works as intended.

This is where the concepts around regression testing strategy come into play even more clearly.

If you don’t know what your entire testing strategy is in the first place it’s going to be impossible to know what kind of testing you need to be employing at each stage of development.

You’ll need to be using different types of testing at different stages and so it is important not to write either regression or functional testing off one way or another.

Importance of regression testing

By now it should be clear the importance of regression testing for your company.

But, just in case it isn’t, let’s examine some examples where more regression (or more complete regression) testing would have revealed holes and avoided calamity.

  • In January 2012 UK retailer Marks and Spencer accidently priced TVs at £199 instead of the £1099 that they should have been listed for. Regression test scripts should have been written to check the price against the database.
  • In 2016 the Nest Thermostat received an upgrade that paralysed devices due to a bug in the firmware. This left customers out in the cold during one of the coldest weekends of the year! Nest should have definitely regression tested the devices before sending out the update.
  • In December 2018, O2 released a software update internally that caused 30 million users to lose access to data. Surely a couple of regression tests could have confirmed the new software would have caused this? (link)

These are just some of the failures that highlight the importance of regression testing (well in some cases any testing at all!)

How important is regression testing for your business?

“In order to improve testing at Blackboard, Ashley and her team didn’t reduce the time it took for the regression tests to execute. Instead, they focused on breaking up the testing into smaller sections so that they were running the most valuable tests at the right time.” Excerpt From: Ronald Cummings-John and Owais Peer. “Leading Quality.”

Realistically speaking, regression testing is only as important as the level of maturity of your product as we’ve previously examined. 

You and your team need to identify this level of maturity and then develop the appropriate testing strategy to match. Here are some example questions to ask (taken from the Leading Quality book):

  • What testing types make the most sense for our product’s current stage?
  • How can we optimize the feedback loops of our testing so that our engineers get the most value from them?
  • Do we need to reach outside our company to external partners to increase the team’s capabilities or capacity?
  • Where can we utilize external partners to fill in short-term or long-term gaps?

We hope that this guide has helped to dispel some of the myths surrounding regression testing!

Keep in mind that your testing strategy, including regression, functional and unit testing, should be constantly adjusted and updated. Think of regression testing as a single pillar which holds up the overall quality of the product that you ship to your customers.

Because, after all, we’re developing software for humans to use and hopefully love.

Need help with QA testing?

We’d love to give you a personal demo of our platform. Find out how we manage, execute and analyse test results to help you release high quality software anywhere in the world.

Ready? Let's talk