Regression Testing

Regression testing is a type of testing that is done to verify that a code change in the software does not impact the existing functionality of the product. This is to make sure the product works fine with new functionality, bug fixes or any change in the existing feature. Previously executed test cases are re-executed in order to verify the impact of change.

Regression Testing is a Software Testing type in which test cases are re-executed in order to check whether the previous functionality of the application is working fine and the new changes have not introduced any new bugs.

This test can be performed on a new build when there is a significant change in the original functionality that too even in a single bug fix.

Regression test is like a verification method. Test cases are generally automated as test cases are required to execute again and again and running the same test cases again and again manually is time-consuming and tedious one too.

Example, Consider a product X, in which one of the functionality is to trigger confirmation, acceptance, and dispatched emails when Confirm, Accept and Dispatch buttons are clicked.

Some issue occurs in the confirmation email and in order to fix the same, some code changes are done. In this case, not only the Confirmation emails need to be tested but Acceptance and Dispatched emails also needs to be tested to ensure that the change in the code has not affected them.

Regression testing is not dependent on any programming language like Java, C++, C# etc. It is a testing method which is used to test the product for modifications or for any updates being done. It verifies that any modification in a product does not affect the existing modules of the product.

Verifying that the bugs are fixed and the newly added features have not created any problem in the previous working version of the software.

Testers perform functional testing when a new build is available for verification. The intent of this test is to verify the changes made in the existing functionality and the newly added functionality as well.

When this test is done, the tester should verify whether the existing functionality is working as expected and the new changes have not introduced any defect in functionality that was working before this change.

Regression testing is usually performed after verification of changes or new functionality. But this is not the case always. For the release that is taking months to complete, regression tests must be incorporated in the daily test cycle. For weekly releases, regression tests can be performed when the functional testing is over for the changes.

Regression checking is a variation of retest (which is simply to repeat a test). When retesting, the reason can be anything. Say, you were testing a particular feature and it was the end of the day- you could not finish testing and had to stop the process without deciding if the test passed/failed. The next day when you come back, you perform the test once more – that means you are repeating a test you performed before. The simple act of repeating a test is a retest.

Regression test at its core is a retest of sorts. It is only for the special occasion that something in the application/code has changed. It might be code, design or anything at all that dictates the overall framework of the system.

A retest that is conducted in this situation to make sure that the said change has not made an impact on anything that was already working before is called Regression Test. The most common reasons why this might be conducted are because new versions of the code have been created (increase in scope/requirement) or bugs have been fixed.

I was just teaching one of these days in my class, and a question came to me – “Can regression be done manually?”

I answered the question and we moved on in the class. Everything seemed OK, but somehow this question nagged me for quite a while later.

Over the many batches, this question comes multiple times in various different ways. Some of them are:

  1. To perform the test execution do we need a tool?
  2. How is regression testing performed?
  3. Even after an entire round of testing– newcomers find it difficult to discern what exactly regression test is?

And of course, the original question:

  1. Can this testing be performed manually?

To begin with, Test execution is a simple act of using your Test cases and performing those steps on the AUT, supplying the test data and comparing the result obtained on the AUT with the expected result mentioned in your test cases. Depending on the comparison result, we set the status of the test case pass/fail. Test execution is as simple as that, there are no special tools necessary for this process.

Automated Regression Test is the testing area where we can automate most of the testing efforts. We run all the previously executed test cases on a new build.

This means that we have a test case set available and running these test cases manually is time-consuming. We know the expected results, so automating these test cases is time-saving and is an efficient regression test method. The extent of automation depends upon the number of test cases that are going to remain applicable over time.

If test cases are varying from time to time, the application scope goes on increasing and then automation of regression procedure will be the waste of time.

Most of the regression test tools are record and playback type.  You will record the test cases by navigating through the AUT (application under test) and verify whether the expected results are coming or not.

Recommended Tool:

#1) Ranorex Studio

Enhance Software Quality and maximize your resources with this powerful automated regression testing tool. You can execute more test cases in a fraction of the time with Ranorex which is up to 78% efficiency increase over manual testing.

Other Tools:

Regression is initiated when a programmer fixes any bug or adds a new code for new functionality to the system.

It is a quality measure to check whether the new code complies with the old code so that the unmodified code is not getting affected. Most of the time the testing team has the task to check the last minute changes in the system. In such a situation, testing only affected application area is necessary to complete the testing process on time by covering all the major system aspects.

This test is very important when there is a continuous change/improvement added in the application. The new functionality should not negatively affect the existing tested code.

Regression is required to find the bugs that occurred because of a change in the code. If this testing is not done, the product might get critical issues in the live environment and that indeed can lead the customer into trouble.

While testing any online website, a tester reports an issue that the Price of the Product is not shown correctly i.e. it shows lesser price than the actual price of the Product, and it needs to be fixed soon.

Once the developer fixes the issue, it needs to be re-tested and Regression testing is also required as verifying the price at the reported page would have got corrected but it might be showing an incorrect price at the summary page where the total is shown along with the other charges or the mail sent to the customer still has the incorrect price.

Now, in this case, the customer will have to bear the loss if this testing is not performed as the site calculates the total cost with the incorrect price and the same price goes to a customer by email. Once the customer accepts, the Product is sold online at a lower price, it will be a loss for the customer.

Given below are the various types of Regression :

  • Unit Regression
  • Partial Regression
  • Complete Regression

#1) Unit Regression:

Unit Regression is done during the unit testing phase and code is tested in isolation i.e. any dependencies on the unit to be tested are blocked so that the unit can be tested individually without any discrepancy.

#2) Partial Regression:

Partial regression is done to verify that the code works fine even when the changes have been done in the code and that unit is integrated with the unchanged or already existing code.

#3)  Complete Regression:

Complete Regression is done when a change in the code is done on a number of modules and also if the change impact of a change in any other module is uncertain. Product as a whole is regressed to check any changes because of the changed code.

This depends upon the scope of newly added features.

If the scope of a fix or feature is too large, then the application area getting affected is also quite large and the testing should be performed thoroughly including all the application test cases. But this can be effectively decided when the tester gets input from a developer about the scope, nature, and the amount of change.

As these are repetitive tests, test cases can be automated so that a set of test cases alone can be easily executed on a new build.

Regression test cases need to be selected very carefully so that maximum functionality is covered in a minimum set of test cases. These set of test cases need continuous improvements for newly added functionality.

It becomes very difficult when the application scope is very huge and there are continuous increments or patches to the system. In such cases, selective tests need to be executed in order to save testing cost and time. These selective test cases are picked based on the enhancements done to the system and the parts where it can affect the most.

  • Re-run the previously conducted tests
  • Compare the current results with previously executed test results

This is a continuous process performed at various stages throughout the software testing lifecycle.

A best practice is to conduct a regression test after the sanity or smoke testing and at the end of functional testing for a short release.

In order to conduct effective testing, the regression test plan should be created. This plan should outline the regression testing strategy and the exit criteria. Performance testing is also a part of this test to make sure that the system performance is not affected due to the changes made in the system components.

Best practices: Run automated test cases every day in the evening so that any regression side effects can be fixed in the next day build. This way it reduces the release risk by covering almost all regression defects at an early stage rather than finding and fixing those at the end of the release cycle.

#1) Retest all:

As the name itself suggests, the entire test cases in the test suite are re-executed to ensure that there are no bugs that have occurred because of a change in the code. This is an expensive method as it requires more time and resources when compared to the other techniques.

#2) Regression Test Selection:

In this method, test cases are selected from the test suite to be re-executed. Not the entire suite is re-executed. Selection of test cases is done on the basis of code change in the module.

Test cases are divided into two categories, one is Reusable test cases and another one is obsolete test cases. The reusable test cases can be used in future regression cycles whereas obsolete ones are not used in the upcoming regression cycles.

#3) Test Case Prioritization:

Test cases with high Priority are executed first than the ones with medium and low priority. The priority of test case depends on its criticality and its impact on the product and also on the functionality of the product which is used more often.

#4) Hybrid:

The hybrid technique is a combination of Regression test selection and Test case Prioritization. Rather than selecting the entire test suite, select only the test cases which are re-executed depending on their priority.

Most of the bugs found in the production environment occur because of the changes did or bugs fixed at the eleventh hour i.e. the changes done at a later stage. The bug fix at the last stage might create other issues/bugs in the Product. That’s why Regression checking is very important before releasing a Product.

Below is a list of test cases that can be used while performing this test:

  • Functionalities which are frequently used.
  • Test cases which cover the module where the changes have been done.
  • Complex test cases.
  • Integration test cases which include all the major components.
  • Test cases for the core functionality or feature of the Product.
  • Priority 1 and Priority 2 test cases should be included.
  • Test cases which frequently fail or recent testing defects were found in the same.

Now that we have established what regression means, it is apparent that it is testing also – simply repeating in a specific situation for a specific reason. Therefore, we can safely derive that the same method applies for testing in the first place can be applied to this too.

Therefore, if testing can be done manually then regression testing can be too. The use of a tool is not necessary. However, as time goes on applications get piled on with more and more functionality which keeps increasing the scope of regression. To make the most of the time, this testing is most often automated.

Given below are the various steps involved in performing this testing:

  • Prepare a Test suite for Regression considering the points mentioned in “How to select Regression Test suite”?
  • Automate all the test cases of the test suite.
  • Update the Regression suite whenever it is required like if any new defect which is not covered in the test case is found, and a test case for the same should be updated in the test suite so that the testing is not missed for the same next time. The regression test suite should be managed properly by continuously updating the test cases.
  • Execute the Regression test cases whenever there is any change in the code, the bug is fixed, new functionality is added, an enhancement to the existing functionality is done etc.
  • Create a test execution Report which includes the Pass/Fails status of the executed test cases.