What is regression testing?
The skill of regression testing is in identifying all unexpected changes before the system is released.
Those deemed as errors can then be removed, thus ensuring the system has not regressed.
What is a successful regression test?
The number of defects found, or maybe the number of test cases ran? The only real measure of success or failure is the customer experience after the software is delivered.
If all is as it should be, and no defects have found their way through, then you have conducted a successful regression test.
Why conduct regression testing?
It is simply to reduce the likelihood of errors in the software adversely affecting the users of that software.
It is a risk mitigation technique and one that is very important in saving companies time and money and the risk of significant embarrassment.
Top Ten Factors for Success
1. Time Window
There would be no point in planning a full system regression that lasted many weeks if the release had to be made tomorrow. The type of development life cycle being used will heavily influence the time window available to regression test a system. If an agile methodology is being used, this would be a much smaller window than if the project is a longer waterfall effort. The efforts required, coupled with the smaller time frames in agile, can be mitigated somewhat by the release train approach.
Depending on the size and scope of change, it may be necessary to use risk-based methods to attempt to regression test in smaller time frames.
2. Pre-requisites
The next factor to consider is the prerequisites for a given test; this will mainly be the people needed to run the tests.
Are specific skill sets required?
Maybe only certain hardware can be used for the tests needed. Even if the tests are all automated, somebody would still need to oversee the run and results. Getting the prerequisites and the time windows aligned is one of the key determining factors of success.
3. Environment
Unless the test environments are ready, then nothing can begin; also, check that the specification of environment-sensitive testing is covered, e.g. localization/language checks. Elements to look for here are ensuring that the build and the data layer are accurately aligned. Many false positives will be given if there are missing new data fields or tables. This is where a smoke test is best adopted; this is a very lightweight run-through of some key processes to check that the environment, database and build are aligned and ready to test.
A good way of thinking about this is if you needed to release the build within a few hours, what would be the tests to run?
These would be the ones that would make the best smoke tests.
4. Data
Usable data for any new functionality may be needed, such as specific set-up data or new types of processes. Standard data for existing functionality and previously found defects must also be available for testing.
This can sometimes be difficult to come by, and new tests in the QA environment will fail due to existing data. This data should either be reusable or renewable if required; obfuscation may also be an issue if the test data is copied from live. The process of creating data may need collaboration with the DBA team, so this should also be taken into consideration.
5. Landscape scope
With regression testing, integration is key. This may be the integration of different components within an application, but also are important across applications. Most systems require end-to-end processes that cross many platforms and databases. It is important to define the landscape to regression test before testing starts.
Sometimes it may be necessary to use stubs or API calls to mimic certain application responses if they are not yet available to use in the test process.
6. Prioritization of risk
After smoke tests are completed, a more thorough set of sanity tests should be run.
These would verify all the important processes, i.e. those that would cause real harm if they went wrong in a live situation.
After these, it is acceptable to investigate the following areas:
- Most used processes
- Areas with recent changes
- Recent defects fixed
- A clean run and negative testing
- Historical areas of defect density
Test cases can be scored based on the above criteria, and the severity/impact technique can give the appropriate risk score. If the test cases are prioritized in this way with a time estimate, it should be easier to plan what can be used to give the best results.
7. Automation
One area of testing that lends itself to automation is regression. Automation allows repeatability and reuse over the more mundane tasks. An automation test can be run hundreds of times with different data and scenarios overnight, ready for the team to review in the morning. One important aspect of regression automation is that it must mimic what the end-user does; therefore, utilizing the UI as opposed to APIs or other unit testing techniques is more vital here.
More efficiency is experienced when the automation can be created quickly and by more of the team. For fast-paced software development environments, this is especially important. Therefore, automation needs to be available to everyone, not just specialized test resources.
8. Script maintenance
With QA personnel changes or in fast-paced development environments, it is essential that automation can be manipulated without any impediments. These impediments normally revolve around the technical aspects of changing code to align with the new applications to be tested. Ease of baselining scripts is very important to allow regression to continue into the next test phase. If waterfall, this may be months away, but if dev/ops or agile, this may be much
closer to hand.
The main reason automation efforts fail into disuse is the lack of people, and time, to correctly realign the automation created to continue to justify the returns needed.
9. Full UI regression
For true regression, all aspects of the software need to be tested; this means that any automation needs to take into account UI changes from previous builds. This full content analysis should be created and checked automatically to give the most effective results. This also allows for compatibility checks across both hardware platforms and operating systems.
10. Output Analysis
QA needs to make the decision as to whether something has passed or failed quickly and act accordingly. To allow this to
happen, the automation being used must create an output that is easy to understand and interpret. It is common for automation to run overnight, so the results can be reviewed in the morning.
Initially, a simple high-level pass/fail is needed after this, though it is important to see exactly where the error has occurred and the steps to reproduce it to see if this is a real software defect or another issue.
Conclusions
Regression testing is a central component in ensuring software quality. Done correctly, the software delivered will be more reliable and robust, increasing in quality more so after every release. Applying the ten key factors above will ensure that your software delivery will be as bug-free as possible every time.
Just make sure when you set things up:
- Your process allows you to identify all changes, including in the UI, not just the expected ones.
- You can quickly and easily baseline scripts to keep pace with the rate of change-you don’t want to resort to writing elaborate code here. If your solution doesn’t support this, it can often fall into disuse.
- Output documents and reports are user-friendly and will make sense to both IT and business users alike.
About Us
At Original Software, our approach to regression testing takes into account over 20 years of experience solving issues and using the key tenants above.
We are proud of our genuinely code-free automation, manual testing assistants, patented UI identification, and baselining self-healing scripts. All of this allows us to offer the only true, out-of-the-box automation regression testing solution on the market today. Got any questions? We’re here to help!
