Continuous Testing vs. Traditional Testing

We hear the term ‘continuous testing’ a lot in the software testing industry. It’s one of those buzzwords that just keeps coming up, time and time again.

But what does it actually mean? And why is it helpful?

Previously, continuous testing was simply seen as the process of executing automated tests as part of the software delivery pipeline.

But, in fact, continuous testing is now being understood as much more than that.

When writing Leading Quality, I spoke to QA experts and thought leaders who shared their thoughts on testing best practices. These conversations painted a clearer picture of how top organisations structure their QA strategy and the techniques they use to achieve quality. 

It was clear to me that continuous testing was top of mind for many of these individuals. 

In this article, I’m going to be discussing the key differences between continuous testing and traditional testing, and the schools of thought in the QA community regarding the benefits of each method. It is my belief that testing continuously can have many interesting benefits, and implementing it effectively can help you achieve quality results. 

So, what does it mean?

Continuous testing does what it says on the tin - it’s all about testing continuously. This means testing software, apps or websites at every stage of their lifecycle. If you are truly executing continuous testing, you are incorporating testing even before a line of code is written, and even beyond release. 

This means you are pre-empting problems before they happen  - putting quality at the forefront early on. Skills you might use to test an application in traditional testing are being used to test a concept, idea, or the very early stage designs that are not fully formed yet. This is similar to the mentality behind TDD (Test-Driven-Development). 

Essentially, you are building testability into your product, and code, right from the very start. 

Dan Ashby wrote an extremely insightful article on this topic, including a fantastic graph to demonstrate what it means to truly test continuously. 

Basically it means… you are always testing! But the testing can, in fact, begin even before the plan. Dan says:

“Testing the idea is vital; investigating to uncover information, refactoring the idea to enhance and solidify it.”

What Dan is suggesting is that the idea of ‘testing’ doesn’t necessarily mean writing an automated test script, or something similar, and sending it off. You can test an idea, discussing its potential faults and bugs, and iron out issues before embarking on a formal, traditional testing process.

Why not just use traditional testing?

Traditionally, QA and testing had both been considered an afterthought - running a series of tests right before release, at the end of the product lifecycle.

What this traditional method means is that the product owner thinks of an idea, decides whether it is viable, sends it off to the dev team to code, who then send it off to the QA team for testing before release.

This sounds simple enough. But in fact, this means the software development process becomes too linear. Rather than a cyclical process, where QA is involved from the very beginning, this traditional method passes the product down the line prioritising speed over quality - and it does mean that bugs appear along the way. 

But, quality can’t suffer in favour of speed of release. 

That’s where continuous testing comes in. 

How to implement continuous testing 

This might sound like a lot more work. And for teams strapped for time and under a lot of pressure to release fast, extra work could seem near impossible. The idea of extra testing at every stage of a product lifecycle could concern even the slickest engineering team going. But continuous testing doesn’t need to slow down your release process, or require extra, expensive internal hires. 

You can implement continuous testing without it becoming a bottleneck. 

So how do you implement this technique effectively?

1. Plan and decide what tests you are conducting very early on
   
   Going into continuous testing without a clear plan will only lead to confusion and wasted time. Think about what tests your product requires at each stage of the development process. This will inform what tools you need to purchase if you need to, and where your team needs to re-focus their efforts. 
   
2. Only test what you need to test
   
   Continuous testing does not mean endless duplicate tests and hours of extra work. 
   
   Break down your release cycle and identify if any unnecessary repetitive tests have been happening prior to implementing continuous testing. 
   
   This will ensure you are not wasting time and money on futile tasks.
   
3. Automate where necessary
   
   This ties in with point two. If you are manually running the same test time and time again with minimal need to change the test scripts: automate it. Automation testing, where necessary, will dramatically speed up your QA process and enable you to run hundreds of tests at the touch of a button.
   
   Test automation provides you with near-instant feedback and works exceptionally well for ‘verifying’ tests. This means you can cut out repetitive work and achieve fast results.
   
   Manual testing techniques all have their place, however, and it is essential in the planning stage you incorporate these in your testing toolkit. 
   
   A blend of automated and manual testing will help you achieve that full test coverage every engineering and QA team needs.
   
4. Set up your tests
   
   Continuous testing can be implemented at every stage of your delivery pipeline. 
   
   So, set up test suites at the key points you identify. These could be every code change, merge or new commit. Use a blend of test methods and start setting up these tests at the very beginning of the coding process, right through to deployment. 
   
   As I previously mentioned, as well as setting up test suites, think about how you can test ideas, build testability into your code, and continue testing well beyond release.
   
   Your continuous testing strategy should be like a living organism: always changing, adapting and building on previous learnings. 

Why testing continuously works

I find research conducted by Capers Jones useful when expressing why this method works.

Jones discovered that, in a typical release cycle, a bug found post-release could cost $16,000 or more to address. That’s enough to make any CTOs eyes water - especially when budgets are strapped.

But, what Jones found is that developers could have fixed that same bug in the early stages for as little as $25. Just take a look at the graph he created below:

What this graph shows is that the highest percentage of defects are introduced in the early stages of the development process. These are the cheaper bugs to fix - so why don’t we fix them earlier on? Why not save precious time, and money, but testing before it's too late?

Continuous testing allows you to catch these frustrating bugs before they become bigger problems. It’s so crucial to think about quality early on, to save you money, unnecessary work, and bugs that cause your team a lot of hassle. 

Conclusion

When done correctly, continuous testing ensures quality in the software you create, reduces critical bugs and saves you money. 

Keeping QA front of mind decreases the likelihood of a huge, expensive bug being discovered right before release date and putting the brakes on the process.

In this way, more testing is not a bottleneck, but in fact, a way to build the foundations for successful future releases.

Quality is crucial. And your customers will always expect it. By testing early, and often, you can improve the quality of your software, and find the critical bugs before your customers do.

Author Bio: After selling his first startup, Ronald Cummings-John is now scaling up Global App Testing, a VC-backed, crowdsourced testing platform with over 40,000+ professional testers globally. Global App Testing was selected as one of the fastest-growing tech companies in the UK. His passion for quality assurance has taken him around the world, and he has worked with QA and product teams from top companies including Etsy, Microsoft, King, Spotify, and eBay. Ronald is co-author of the book, "Leading Quality: How Great Leaders Deliver High-Quality Software and Accelerate Growth.