What is Regression Testing? | Methods & Benefits

What is regression testing? What regression testing tools are available? How do you perform regression testing? We're here to answer all your burning questions in our ultimate guide.

 

 

development-team-four@2x

Introduction

Regression testing is among the most important parts of your overall testing suite and among the most well-known types of test. Here’s our breakdown of the best practices, the failure categories, and how to get started with regression tests. If you want to get started with regression testing with Global App Testing, you can do so here.

Recap | What is regression testing?

Regression testing is a type of QA software testing that ensures changes or updates to an existing software product do not affect previously functioning features. A regression test asks, “does everything still work as expected?” A malfunction or bug in another part of the system caused by a new release is called a “regression” – hence, “regression testing.” 

A dev team is generally expected to run a regression test suite at the end of a development sprint, but they can also perform regression testing during development as part of your CI/CD pipeline in an agile testing environment. How often you should execute regression tests depends on the incentives for your business and the difficulty of the test.

Speak to a QA Expert

Speak to a QA Expert

Why is regression testing important?

What regression tests identify – that your existing functionality continues to work as expected – is likely to be extremely important for your business, making regression testing an important defence against against severe risk. 

It can also surprise unseasoned developers to see how likely that risk is to occur. Even tiny changes to the code can have outsized and unexpected impacts on completely separate elements of the product. So the risk frequency is higher than you might expect.

It’s also much less severe outcome for the business to release broken new functionality than it is to break things upon which people already rely. As businesses shift from startup to scaleup, it becomes more impactful to prevent regression than to add new features quickly. To break an existing feature is massively disruptive, and will cause a huge headache for your CS team. So the larger your user base, the more important regression testing becomes.

Regression testing is also one of the most common types of test your business conducts, meaning that it’s also important to think about regression testing in operational terms. Done wrong, it can stack up significant cost to your business and can slow your CI/CD cycle down.

How is regression testing different from… re-testing, unit testing, functional testing 

Re-testing

Retesting is where you search for a specific defect you’ve recently found (and perhaps fixed). Regression testing is where you search generally for a defect across the product following a new feature release.

As per unit and functional testing; regression testing isn’t a technique, but an outcome, so all kinds of different tests are appropriate to include within a regression testing suite. That’s almost certain include functional tests and unit tests – and the majority of regression tests are functional test case execution – but might also include any other kinds of test.

Automatic program testing diagram

Courtesy: Keizo Tatsumi

How to prioritize your test cases for a regression test 

One of the methodologies for regression testing is “retest all” – which is what it sounds like, a complete re-execution of all of the test cases on your system. This is time intensive and expensive. Instead, businesses are more likely to choose a selection of test cases and execute on them.

How many test cases?

It is not possible to answer “how much” regression testing you should do and how often, (although as a testing company, we generally argue in favour of a proactive and thorough testing approach).

Instead, we’d recommend identifying two or three distinct regression suite tiers and calculating their cost and impact level. Ensure you can calculate both the time and the cash cost of executing each tier of test suite so that you can refer quickly to your figures at the decision stage. This will enable your business to choose the appropriate depth for the moment and encourage you to keep track of what level of testing you are conducting in general – and whether that aligns with your budget and risk appetite. 

What kind of test cases?

Once you have made the decision not to “retest all”, you will be confronted with the question of what test cases to prioritise in a given regression suite. 

Again, this is often specific to your own business and product. But we’ve seen our clients use the following values to prioritize test cases in their regression suites. 

  • High vulnerability test cases, for areas of your product which are prone to bugs and imperfections 

  • Critical test cases, to rule out the most important failures, from a safety, legal, or business perspective

  • Complex & integration test cases, where integrations are simultaneously business critical and prone to failure 

  • How relevant those cases are to a particular login or release / e.g. login test cases for login system change 

  • A variety of test cases. To ensure that all parts of the product are tested, a selection of test cases which together cover most system modules are selected.

  • Randomly selected cases.  By revolving through test cases, you can ensure that over multiple regression suites you are undertaking a full system test. 

  • Some combination of the above.  Most frequently, regression test cases are a reflection of multiple different priorities. As a compromise between time, risk, and different priorities, regression suites are about judgement. 


Thinking about automation in regression testing

Regression testing can be done manually or with the help of automated testing tools. 

Manual regression testing involves manually retesting the software after changes have been made. This can be time-consuming and may require a large team of testers. Automated regression testing, on the other hand, uses software tools to retest the software automatically. This can be faster and more efficient than manual testing, but it requires specialized skills and expertise to set up and maintain the testing tools.

Developers tend to like the idea of automating tests, and as one of the biggest test categories, regression testing is one of the biggest targets.

 But there are challenges to test automation:

  • Businesses generally automate less than they expect. Research from TestRail shows that businesses tend to overestimate how far they will automate their testing suite. In their annual survey to the same respondents, TestRail asked users the % of automated tests in their total test suite and the % they expected to have automated by the end of next year. But despite estimating reliably that they would automate 30% more of their tests next year, the real average increase was more like 1%. It reflected not just a failure to automate tests, but a failure to recognise they were not making forward progress. (Check out our webinar) 
  • Flaky tests. In a webinar conducted by TestRail and Global App Testing, respondents identified flaky tests as their biggest barrier to automating more. The half-life of automated tests is short and we’ve devised a formula for identifying whether it’s appropriate to automate a given test case. 
  • Time investment is a vicious cycle. The failure to automate tests creates a new time demand for an engineer: manual testing. That can be a vicious cycle as less time spent automating or fixing can lead to more time spent manual testing which can lead to less time spent automating - and so forth. 

Global App Testing is a crowdtesting platform - we send tests to a network of human testers around the world (like “Uber for testing”) and you receive them back in less than 5 or less than 48 hours, depending on the level of specialism.

That means that if you’re struggling to try to automate more of your tests, Global App Testing lets you simulate the speed benefits of automated testing without demanding the automation work from your engineers. In fact, we actively help businesses close their “automation gap”.

Get started →

 

Regresstion testing strategy by Dan Ashby

Source: Dan Ashby

Best practices & failure categories for regression testing

Best practices 

  1. Give yourself a buffer day. Regression testing is often time-intensive and planning around it can be poor. Other items eat into your testing time; and you have less time than you budgeted for. That makes it a good place to have a buffer day, so that you can eat into the buffer day rather than release something you’re not satisfied with into production.

  2. Test in an environment which is as close to production as possible. The closer to the production environment you can undertake while still testing early in the process the better. In particular, the size of the database the test will interact with may have results when it comes to 

  3. Know when to cancel.  If you have a two day regression cycle, after the first four hours you know there’s too many problems for the build to “pass” the test, you’ll be faced with whether to complete the test and find all the issues

  4. Define the scope: Clearly define the scope of the regression testing process to ensure that all necessary features and functionality are tested.

  5. Automate whenever possible: Use automated testing tools to speed up the regression testing process and reduce the risk of human error.

  6. Prioritize test cases: Prioritize test cases based on their criticality and potential impact on the software.

  7. Use real-world scenarios: Test the software using real-world scenarios to ensure that it meets the needs of end-users.

  8. Reuse test cases:  Reuse existing test cases to save time and resources.

  9. Before a big test case, start with a smoke test to prevent additional resources being wasted.

Common failure categories of regression testing 

  1. Incomplete testing: When not all features or functionality are tested, the regression testing process can be incomplete, leading to potential issues in the software.

  2. Insufficient test coverage: Insufficient test coverage can lead to missed bugs and other issues, which can impact the overall quality of the software.

  3. Lack of prioritization: Without prioritization, test cases may not be executed in the right order, leading to delays and potential issues in the software.

  4. Inadequate reporting: Inadequate reporting can lead to a lack of visibility into the testing process and the results, making it difficult to identify issues and track progress.

 

Regression in agile 

In an Agile or DevOps workflow, software development teams must balance the need for rapid releases with the need for thorough testing. Regression testing can be a bottleneck in this process if not managed properly. However, there are ways to regression test effectively without slowing down the development process. In this section, we'll explore how you can regression test in an Agile or DevOps workflow without it becoming a bottleneck.

 

1. Automate Your Tests

One of the most effective ways to speed up regression testing is to automate your tests. Automated testing tools can help you execute test cases quickly and accurately, freeing up your team to focus on other tasks. Automated tests can also be integrated into your DevOps pipeline, allowing you to catch issues early in the development process.

 

2. Prioritize Your Tests

Another way to avoid regression testing becoming a bottleneck is to prioritize your tests. Prioritizing your tests based on their criticality can help you focus on the most important features and functionality, ensuring that they are thoroughly tested. This approach can also help you save time and resources by avoiding testing less critical features and functionality.

 

3. Use Parallel Testing

Parallel testing involves executing multiple test cases simultaneously, which can help speed up the testing process. This approach is especially useful for large-scale regression testing, where executing tests sequentially can take a significant amount of time. Parallel testing can be achieved using automated testing tools and can help you catch issues quickly and efficiently.

 

4. Embrace Continuous Testing

Continuous testing involves testing throughout the development process, rather than just at the end. This approach can help you catch issues early and avoid regression testing becoming a bottleneck. By integrating automated testing tools into your DevOps pipeline and continuously testing your software, you can ensure that your software is reliable and bug-free.

 

Regression testing tools 

There are a large number of tools available on the market to help you undertake a regression test.

  1. Global App Testing: GAT’s dashboard allows you to send a test to 70,000 testers around the world with a 6 hour turnaround, meaning that it complements the below automated testing tools perfectly.

  2. Selenium: It is a popular open source test automation tool used for testing web applications. It supports multiple programming languages and can be used for functional, regression, and performance testing.

  3. Appium: It is an open-source tool used for mobile app testing. It supports both Android and iOS platforms and can be used for functional, regression, and performance testing.

  4. JUnit: It is a popular open-source testing framework for Java-based applications. It is mainly used for unit testing but can also be used for regression testing.

  5. TestNG: It is a testing framework that supports both unit testing and regression testing. It is widely used for testing Java applications.

  6. Visual Studio Test Professional: It is a commercial tool that can be used for functional and regression testing. It supports multiple programming languages and platforms.

  7. HP Unified Functional Testing (UFT): It is a commercial tool that can be used for functional and regression testing. It supports a wide range of technologies and platforms.

  8. Apache JMeter: It is an open-source tool used for load testing, functional testing, and regression testing of web applications.

Need help with QA testing?

We’d love to give you a personal demo of our platform. Find out how we manage, execute and analyse test results to help you release high quality software anywhere in the world.

Ready? Let's talk