Is Software Testing Really Worth It?

Yes, and no. Well, as with everything, it depends. In this post, I'll try to illustrate my experience with testing as a developer, then as a startup founder.

About me

Hi! I'm Simone, a full-stack developer, and CTO with 15 years of experience coding in various industries, including web agency, e-commerce, adult, and enterprise software. In this blog post, I'd like to share insights on software testing and how to determine appropriate methods for different business needs and stages.

I'm the CTO and Co-founder of Bugpilot, a bug monitoring tool that helps SaaS teams catch bugs that slipped through the development, testing, and QA processes.

Introduction

Different kinds of testing

In the software development world, many testing practices have long been regarded as essential for ensuring the reliability and quality of software products. However, it is important to recognize that while these methodologies have their merits, they are not without their drawbacks.

This blog post aims to spark a conversation and shed light on the potential harm that TDD and QA can inflict on software teams. By acknowledging that there is no one-size-fits-all approach and that different businesses and stages require appropriate methods, we can begin to navigate the complexities of software development more effectively.

I acknowledge many different methodologies, common practices, and recommendations around software testing; in this post, I focused mostly on Test-driven development and Quality Assurance.

Here's an overview, in case you're not familiar with the terms:

TDD

Test-driven development (TDD) is a software development method that involves writing automated tests before writing code. This helps catch issues early in the development cycle, ensuring that the code is thoroughly tested and meets business requirements. Developers first define the expected behavior of a new feature with a test, then write the code to pass the test, and finally optimize the code through refactoring.

QA

Quality Assurance (QA) ensures software meets quality standards before release. Typically a manual process carried out by a QA Engineer, identifies and fixes defects through manual and automated testing, such as verifying functional requirements and checking performance. This ensures reliable, stable software that meets business and end-user needs.

A couple of questions for you!

My goal with this post is to spark a conversation. I can't stress enough that I strongly believe that tests are needed in many cases. Think of medical appliances, aircraft software, control systems, banking, and much more, software in use by billions of people where the color of one button makes a significant profit difference.

Software development requires a balance between structure and flexibility, and blindly following testing principles without considering the context can lead to suboptimal outcomes - code libraries probably benefit from 99% coverage. But does your software need 99% coverage as well?

I prepared a couple of questions. Try to answer yes or no — why is that?

• Do you really need to test your UI components?
• Do you need E2E testing for that “update profile picture” functionality?
• Do you fail QA and delay a release if there's a minor inconsistency between Figma designs and actual UI? What's the boundary?
• Should I fire John for refusing to write tests?

Chapter 1. My case against Test-Driven-Development

Challenging the dogma

Test-driven development (TDD) has gained an almost-religious following. Some claim it helps with mental health, and it is often accompanied by the dogmatic belief that it is the only path to success. This chapter aims to explore the drawbacks of TDD and shed light on the rigidity trap, time constraints, and false sense of security.

As a CTO, I have encountered colleagues who staunchly insisted on TDD for even the most trivial functions. The prevailing dogma seemed to be, "You must do TDD, or you're an idiot!" This unwavering belief in TDD as the only acceptable approach often made me question my own judgment and competence. I distinctly recall incidents when I was being laughed at for refusing to write tests for a simple 2-line function to filter an array.

This personal experience highlighted one of the key pitfalls of TDD—the rigidity trap. The dogmatic adherence to writing tests first can sometimes lead to rigid processes, which may, in turn, impact a company's time-to-market and competitor advantage.

Is it really worth the time?

A significant challenge posed by TDD is the time constraint it imposes on the development process.

Writing comprehensive tests for every piece of code can be time-consuming, especially for trivial functions that have minimal impact on the overall system. In fast-paced environments where quick iterations and timely delivery are crucial, the additional overhead of TDD can slow down the development cycle and hinder agility.

I find it unreasonable to always go with the “I need 99% coverage” approach. Balancing thoroughness and efficiency is essential, and there are scenarios where a more streamlined testing approach may be more appropriate, allowing for faster iterations and quicker responses to market demands.

Moreover, the complexity and imperfection of writing tests that involve interactions with external dependencies or complex systems can further compound the time constraints of TDD.

As a developer, I might even say I enjoy writing tests, but… find a CEO who's happy their team spent 80% of their time writing tests for code that will eventually be deleted.

Developers often need to create mock objects or stubs to simulate the behavior of these dependencies during testing. While mocks can be valuable tools for isolating code and reducing dependencies, they can also introduce additional overhead and complexity.

The reliance on mocks may lead to an imperfect representation of real-world interactions, as it can be challenging to replicate the behavior of external systems or third-party components accurately. This can introduce a level of uncertainty into the testing process, potentially resulting in false positives or false negatives.

Tests are passing; I can have a good night's sleep, right? Right?

Tests are passing; I can sleep safe and sound!

There's an inherent but often overlooked danger of relying solely on tests—a false sense of security. While testing is undoubtedly essential for identifying and preventing defects, it is not a foolproof solution.

“Did we test on every device?”

It's also difficult to test all the possible cases. Especially in the realm of web development, there is a multitude of factors that can impact the user experience. One such factor is the diversity of end-user devices, including different screen sizes, resolutions, operating systems, and browsers. Each combination presents a unique set of challenges that can affect how the software functions and appears to users.

Consider the vast array of devices and configurations: smartphones, tablets, laptops, and desktops, running on iOS, Android, Windows, or macOS, using various versions of browsers like Chrome, Safari, Firefox, Edge, or Internet Explorer. Each device, operating system, and browser combination may render web content differently, and user interactions may vary as well. It is virtually impossible to anticipate and account for every possible device and configuration, making it challenging for tests to provide complete coverage.

“Did I test this UI as if I were my grandma?”

User personas and profiles add another layer of complexity. Your software may target a diverse audience with different preferences, behaviors, and accessibility needs. Testing can help uncover issues that arise in typical scenarios, but it may miss edge cases or specific user interactions that deviate from the norm. For example, users with visual impairments who rely on assistive technologies may encounter usability challenges that are difficult to capture through automated tests alone.

In addition to the technical variations, user context and real-world scenarios play a crucial role in shaping the user experience. Factors such as network conditions, bandwidth limitations, or concurrent usage can impact the performance and reliability of the software.

While tests can provide a controlled environment, they may not accurately simulate the diverse network conditions and usage patterns that users encounter in their daily lives. If tests cannot ensure your software works well, are they worth it at the end of the day?

Chapter 2. QA & The need for speed

I've witnessed firsthand the challenges posed by “strict” Quality Assurance practices, particularly when it comes to smaller companies that must move swiftly to stay on top of their competitors.

I believe this is especially true for early-stage startups; you're somehow likely to throw away the whole feature soon if your customers are not using it. So, was that one whole week of testing and refining the unit tests worth it?

Let me share my personal experiences and shed light on the dangers of getting caught up in perfectionism, especially when minor visual or presentational aspects become the focus of QA efforts.

In my previous role at a startup, we faced a constant battle between delivering features quickly and ensuring impeccable quality. There were instances when our release cycle was unnecessarily delayed due to minute issues such as a misaligned CSS margin, an incorrect font choice, or a missing line break. While attention to detail is important, I began to question the impact of obsessing over these cosmetic imperfections on our ability to stay ahead in the market.

One of the risks of excessive QA is the prioritization of perfection over practicality. In our pursuit of flawlessness, we often found ourselves investing significant time and resources into fixing minuscule visual glitches. Isn't this the enemy of productivity?

While it's essential to maintain high standards, I started to realize that dedicating extensive effort to these minute details might be counterproductive, diverting our focus away from core functionality and user experience improvements that truly mattered to our customers.

The danger became apparent as we observed the consequences of an overly cautious QA approach. Our team began to exhibit risk-averse behavior, opting for a slow and meticulous release process. While the intention was to deliver a near-flawless product, we inadvertently stifled innovation and hindered our ability to respond quickly to customer demands. As a small (30 employees) company, we relied on our agility to iterate rapidly and outmaneuver larger competitors. However, excessive QA practices resulted in a widespread fear of “bugs” which was holding us back.

Shouldn't we accept bugs exist? Anyone who works makes mistakes; if you don't work, you never make mistakes. (Sorry - football quote)

Chapter 4. Customers might be the best testers

The financial implications of prolonged-release cycles became evident. Missed market windows, delayed revenue generation, and potential customer churn began to take a toll. We couldn't afford to dawdle as a small company with limited resources. We needed to leverage our agility and speed to seize opportunities and maintain a competitive edge.

The time spent on perfecting minor details needed to be balanced against the need for rapid iteration and market responsiveness. Postpone, postpone, and postpone every release until we are “more certain” that everything works; shipping on Tuesdays became the preferred option. Why ship on Monday if we can take one more day to test everything?

While testing helps identify and prevent defects, it is equally important to embrace user feedback as a valuable source of information. Users' experiences, preferences, and suggestions can provide insights that go beyond what testing alone can uncover. We can create a user-centric product that meets their expectations by fostering a feedback loop with users, actively listening to their needs, and incorporating their input into the development process.

Rather than relying solely on extensive internal testing, involving users in the testing process can be highly beneficial. Early user testing, such as beta testing or usability studies, allows for real-world scenarios and user interactions to be observed.

This user-centric approach helps identify pain points, usability issues, and unanticipated problems that may not be caught through internal testing alone. Incorporating this feedback early on can greatly enhance the user experience and satisfaction.

Unfortunately, I personally witnessed a strong “unbalance” in many software teams. Here's a question for you: Should QA fail for a UI inconsistency? How often should QA fail?

Chapter 5. “Users will leave if they encounter a bug!”

Research consistently highlights the negative impact of broken functionality on user retention. We have been told many studies show that users are less likely to continue using a product if they encounter frequent bugs, crashes, or errors that disrupt their experience.

According to a survey by Qualitest, 88% of users will abandon an app after just one or two instances of poor performance. These findings emphasize the critical role that functional stability plays in retaining users and building long-term engagement.

When users encounter broken functionality, it erodes their trust in the product and the development team behind it. Even if the issues are eventually resolved, users may develop a negative perception and be reluctant to return. Users may lose trust in a website or app if they repeatedly experience functional errors or glitches.

But… is this still true in 2023?

Bugs are everywhere, and we all know bug-free software may never exist.

Users may encounter bugs from time to time, and it is important to distinguish between minor bugs and critical functionality issues. Users may be nowadays more forgiving of minor bugs that do not significantly impact their overall experience. In our view, users have learned to accept bugs as inevitable in software development; bugs are part of our daily lives.

However, when it comes to broken core functionality that hinders their ability to use the software as intended, users are likely to be less forgiving and may seek alternatives.

B2B bugs are more painful (or not?)

In B2B scenarios, broken functionality can have severe consequences for users and their businesses. It can result in delayed project timelines, missed deadlines, and even a $440 million financial loss.

As for consumer software, business users might become frustrated and angry when encountering bugs preventing them from accomplishing their work. Their loyalty to a software product is closely tied to its reliability and ability to help them succeed in their professional responsibilities. Low reliability should equal higher churn.

However, I have learned that's not always the case. Once a whole organization adopts a technology, is it that easy to make the whole company switch to a competitor solution? Unlikely.

E-commerce has (more) loyalty challenges.

In e-commerce, users have numerous alternatives readily available at their fingertips. As a user, your job-to-be-done is very clear. You need to buy a vacuum cleaner. If the Amazon app crashes, how long before you try the next app in your drawer?

Broken functionality or bugs can lead to quickly abandoned shopping carts, permanently decreased customer satisfaction, and lost business opportunities.

Chapter 6. Are TDD & QA the only solution?

Obviously, no. While tests play a crucial role in identifying and preventing some defects, there are additional measures that can be taken to minimize the impact of bugs on users. Here are a few approaches I have studied:

Monitoring and Error Tracking

Implementing robust monitoring and error tracking systems allows for proactive identification and resolution of issues. Real-time monitoring can help detect anomalies, performance issues, and errors, allowing for prompt remediation before they impact multiple users. Error tracking enables the capture of error details and helps prioritize bug fixes based on their impact on users.

Tools such as Sentry, Rollbar, Bugsnag, and Bugpilot help teams automatically detect coding errors and problematic UX sessions, so teams can proactively address issues.

User Feedback and Support

Actively encouraging and collecting user feedback provides valuable insights into usability issues, bugs, and areas for improvement. Promptly addressing user-reported issues and providing support demonstrates a commitment to resolving problems and maintaining a positive user experience.

Tools such as Canny, Hotjar, and Bugpilot help teams easily collect feedback from their users.

Documentation and User Education

Clear and comprehensive documentation and user education materials can help users navigate the software effectively and minimize the risk of user-induced errors. Providing resources that explain common issues and troubleshooting steps empowers users to resolve minor problems independently.

At Bugpilot, we use a Notion page to keep all our documentation. Easy and free.

Conclusion

Testing acts as a crucial preventive measure by identifying issues early in the development process. Thorough testing, including unit tests, integration tests, and end-to-end tests, helps - to some extent - catch bugs and ensure the stability and functionality of the software. But over-testing and too strict processes are very likely to damage the company in the long run.

However, errors may occasionally slip through the testing net despite our best efforts. Therefore, it is equally important to have mitigation strategies in place. By implementing mitigation solutions, software teams can quickly detect and address errors, minimizing their impact on users and swiftly resolving any issues that arise.

Recognizing that no software might ever be entirely bug-free, creating an environment that encourages user feedback and provides effective customer support is essential. By actively listening to user reports and promptly addressing their concerns, software teams can maintain a positive relationship with their users and foster trust and loyalty.

tl;dr

Don't overtest. You probably don't need 99% unit test coverage. Ship fast.