..and what does it have in common with Spot the Difference?
If you say a word out loud often enough, it loses all sense of meaning. And, sometimes, if a term has been floating around an industry for long enough, it can be easy for people, or even entire companies, to forget what those terms mean – and why they exist.
There are lots of blogs floating around the internet titled “what is regression testing”, “introduction to regression testing”, and similar things. Some of them are even pretty good. So, in this blog, we aren’t going to patronize you by dwelling on definitions (though we will cover them in case you’re new to the software testing game). Instead, we’ll focus on the bit you really care about – how to make sure YOUR regression testing is as good as it can be.
A definition of regression testing
Ready? Here you go:
Regression testing is testing you do to check that everything still works as expected after you’ve changed something in your software (such as after an update).
That’s it. You can go and click around some other blogs if you like, but that’s what it is. If you’re curious, it’s called regression testing because we’re making sure the software hasn’t “regressed” (AKA “got worse”) as a result of your changes.
In fact, we’ll go one step further in simplifying the matter: at its heart, regression testing is like a big game of spot the difference. When you run a regression test, you’re basically using the new version of your software and trying to work out everything that’s changed compared to the old version.
Why is regression testing important?
If you’ve ever tried writing software, you’ll know that it’s a nightmare trying to work out what changing one part of an application will do to other parts of it. An improvement to the UI over here could have disastrous consequences for a piece of vital functionality over there.
Scaling that up to an enterprise setting, a change in one piece of software might not just affect how it performs – it might affect how a connected piece of software performs. Perhaps a piece of data that gets sent from application A to application B is now in the wrong format, and application B can’t make use of it. Perhaps application A now can’t send anything at all. Who knows? As the number of applications that businesses use and the relationships between them grow ever larger and more complex, it’s basically impossible to predict the impacts of software changes on your network of applications.
So, instead, we use regression tests to make sure that everything still works as it should after a change.
Why should I do regression testing?
Put simply, regression testing can save you:
- userTime in later test stages
- Complication in later test stages
- Lost revenue
- Reputational damage
- Lawsuits
Some of those sound a bit dramatic, don’t they?
If you don’t do regression testing, then you will have more bugs slipping through – either to later test phases or straight into production if there are issues that tests such as user acceptance testing (UAT) won’t pick up.
More bugs being picked up by your users during UAT drags out the whole UAT process – which is already long, expensive, and often frustrating for everyone involved.
And, of course, more bugs entering production runs the risk of your business suddenly grinding to a halt or falling over. The web is littered with cautionary tales of companies losing millions in revenue because they can’t operate for a day or of customers unable to access key services due to IT glitches – even of lawsuits and fines being handed down to CIOs over the issue.
When you consider that regression testing is an ideal candidate for test automation, due to the repetitive nature of the testing, you would – quite frankly – have to be mad to skip regression testing.
How to do regression testing well
OK. Here we go. The interesting bit. Because, of course, regression testing that’s done badly doesn’t get you much further ahead than if you’d not bothered at all – and there are some major challenges with how most of the world does regression testing.
First things first: move away from prescriptive testing.
Most regression testing solutions ask you to specify what you’re checking for – whether that’s ensuring a calculation is correct, that key elements such as buttons are present, or that application data such as an order number has been attached to the data being processed. Given that we’ve just discussed how unpredictable the effects of software updates and changes can be, I’m sure you can see the problem with this approach. It’s the testing equivalent of losing your keys and someone saying, “where was the last place you had them?” It’s well-intentioned but utterly useless, and it’s also incredibly frustrating.
This is where our “spot the difference” mentality pays off. Our regression testing solution doesn’t ask you what you want to test for. It just shows you everything that’s changed, from broken links to missing buttons to spelling errors. Some of those changes won’t matter, of course, but the solution highlights the ones that we think will matter – meaning you can catch errors you’d never have spotted with a prescriptive regression test.
Second: use tools that work across your entire stack
In this day and age, nobody is using just one system or ecosystem. Most enterprises use applications from a variety of vendors, with a range of on-premise and cloud-based implementations. There will be custom applications (often performing vital roles in the operation), and you may find 30-year-old IBM green screen terminals jostling side-by-side with cloud-based Salesforce implementations. The trick is using as few tools as possible to test all these.
At Original Software, we’re proud that our solutions will test anything and can even test processes that span multiple systems. So, your test managers only have one tool to learn and one library to maintain. Speaking of which…
Third: minimize manual library changes
If you’re familiar with regression testing, then you know all about baselining. If you’re not, then baselining is where you run your tests on software that you know works. Scientists might call it their control group.
Those familiar with regression testing may also know the pain of having to update that baseline in their test library. Every time you reach a new, stable version of the software that works well in your environment, you really should record a new set of baseline tests to compare the next set of tests to. In a world where cloud vendors can release new updates every month, that can lead to a lot of time, energy, and effort being spent on an activity that isn’t even really testing.
Ideally, your regression testing solution would be able to automatically set a new baseline from any of the tests that you’ve run. So, once your new software has passed a test, you make the new baseline in one click.
At this point in the blog, would it surprise you if I said that our software can do that?
More information about (our) regression testing
Original Software has helped hundreds of enterprises get regression testing done better, faster, and with less effort. Over the years, we’ve created some handy resources for those looking to learn more about the art of regression testing, or to educate others in their organization:
What killed regression testing? (Webinar)
Reviewing key areas where regression testing failures happen and how to avoid those issues, this webinar from 2021 is still relevant today.
4 challenges with regression testing (blog)
In this blog, we cover some of the topics we’ve touched on in this one but in more detail.
Top ten factors for a successful regression test (article)
If you’re the kind of person who absorbs information best in listicle form, this is the asset for you. Find out how to keep your regression tests performing their best in this handy article (with downloadable PDF!)
CertainTeed story (case study)
An example of regression testing done right. CertainTeed used our automated testing and test management solutions to completely stop all bugs from reaching production.
Alternatively, visit our dedicated regression testing solution page.
Of course, if you’re interested in making some changes to how your regression testing works or you’re just building out your regression testing capabilities, you should really have a look at our test automation solution. It’s able to do everything we’ve discussed in this blog, helping you reap the business benefits of best-practice regression testing. That includes:
- Spot the difference testing, where you see EVERYTHING that’s changed – down to spelling, imagery, and system performance.
- Self-healing scripts that can continue to run even if your software has changed significantly and automatically update themselves for the next test.
- Application-agnostic, end-to-end testing, so whatever activity you’re testing, on whatever application, you’ll always be able to use our tools.
- Effortless baseline and test library maintenance to keep admin to a minimum, even if software changes are coming thick and fast.
To find out more, just click here to get in touch with us.
