Why Testing in Production Will Transform Your Success Game

Let's start with a simple question: Why is QA in Production Necessary Despite the Presence of Preproduction Environments?

1. Closing the Gap: Despite the utmost efforts to replicate production environments, preproduction environments can never be a perfect match. Minor variations in configurations, data, or external integrations can exist, leading to unexpected behaviors when the software goes live. By conducting tests in the actual production environment, developers gain real-world insights and identify potential issues that might not have surfaced during preproduction testing.

2. Real Data, Real Results: Preproduction environments often utilize synthetic or sanitized data, which may not accurately represent the variety and volume of data found in the live environment. Testing in production allows QA to work with authentic data, providing a more realistic assessment of the application's performance, scalability, and reliability. This ensures that the software can handle the actual data volume and usage patterns it will encounter in production.

3. Handling Complexity: Production environments can be significantly more complex than preproduction environments due to increased user traffic, third-party integrations, and live data feeds. These complexities can introduce unique challenges and vulnerabilities that may remain hidden until the software is deployed to the live environment. By conducting tests in production, QA can validate the software's behavior under these intricate conditions and make necessary adjustments.

4. The Lack of Monitoring in Preproduction: Monitoring is crucial in various fields. In technology and software, monitoring refers to observing and tracking multiple metrics and performance indicators of systems, applications, networks, or services in real time or over time. The primary goal of monitoring is to ensure that everything is functioning as expected, identify potential issues, and take proactive measures to maintain optimal performance and reliability.

   
   

   Examples of Proper Monitoring Include:

   1. Website Performance Monitoring.
   2. Critical business scenarios (amount of registration, amount of any transactions, placing new orders, and so on)
   3. Server Infrastructure Monitoring.
   4. Application Performance Monitoring (APM).
   5. Network Monitoring.
   6. Security Event Monitoring.
   7. Database Monitoring.

In some cases, resource limitations might be in establishing and upkeep a monitoring system in the preproduction environment. Nevertheless, the primary challenge lies in the need for real traffic and usage in the live production environment.

Beneficial QA practices in production testing!

1. Gradual Rollouts: Implement a gradual rollout strategy to limit the impact of changes. Start with a small percentage of users or traffic, and gradually increase as you gain confidence in the changes.

2. Feature Flags: Use feature flags to enable or disable specific features in the production environment. It allows you to turn off a feature if any issues arise quickly.

3. Monitoring and Observability: Implement robust monitoring and observability tools to monitor the system during testing closely. Real-time data helps quickly identify and address any anomalies or performance issues.

4. Automation: Automate the deployment process to ensure consistency and reduce the chances of human error.

5. Fallback Plan: Always have a well-defined rollback plan in case issues arise. Being prepared for unexpected scenarios is crucial for maintaining system stability.

6. Checking Completed Tasks to Guarantee Smooth Deployment and Proper Functioning in the Production Environment: For instance, when deploying a new feature, once it is deployed to the production environment, the QA team should verify that the feature has been successfully released. To accomplish this, conducting a happy path scenario would be enough.

7. Smoke Tests in Production: Smoke testing is a type of preliminary testing aimed at determining whether the basic functionalities of an application work as expected after a new deployment. In a preproduction environment, smoke tests are traditionally performed to ensure that significant functionalities are not broken before moving forward with more comprehensive testing. However, smoke testing in production takes this approach one step further by validating the core functionalities directly in the live environment after the deployment has been rolled out.

8. Effective Logging: When issues arise, logs provide a detailed record of what occurred, allowing developers and QA to trace the sequence of events leading to the problem. By analyzing the log data, developers can pinpoint the root cause of errors, leading to faster and more effective problem resolution. The QA team must verify that the new features are accompanied by appropriate logging. If they still need to, they should promptly notify the developers about the need for necessary enhancements.

   
   

Finally, I would like to share with you the outcomes and the practices that have been implemented in my company.

Here is the list of them:

• Feature Flags - We have recently started applying this approach, which is why it is too early to discuss the outcomes.
• Monitoring and Observability - QA plays an active role in the monitoring process. While writing System Requirement Specifications (SRS), QA examines which technical metrics, like method execution duration and event occurrence, could benefit the feature. Moreover, during the Business Requirement Specification (BRS) phase, just before the feature implementation, we keep an eye on business metrics, such as the number of users adopting the new feature. We promptly alert the feature owners if we do not find such metrics in the feature documentation during development.
• Automation CI/CD - Our company has implemented a completely automated deployment process to minimize potential human-related errors.
• Fallback Plan - Before rolling out any updates, the developer must assess the feasibility of a seamless rollback to the previous version in the event of widespread issues occurring in the production environment. Following each release, the assigned developer and the DevOps team diligently monitor logs and metrics for any irregularities in our established process. If any significant anomalies are identified, we immediately initiate the rollback procedure.
• Checking Completed Tasks to Guarantee Smooth Deployment and Proper Functioning in the Production Environment - Following the release, developers keep an eye on metrics and logs. At the same time, QA ensures the successful deployment of new features by conducting smoke tests on the implemented functionality.
• Smoke Tests in Production - We categorize test cases into smoke, regression, full set, and feature-specific sets. After the production release, we automatically run production smoke tests, covering the system's most critical functionality. If a test case still needs to be automated, the tester responsible for the release performs it manually.

Over the past four months, smoke production tests have uncovered three critical bugs. Among them, two were linked to discrepancies in settings between preproduction and production environments, particularly related to external providers. The third bug was attributed to the production environment having a different number of databases.

While it's impossible to catch all errors in preproduction, efforts should still be made to minimize them. Adhering to the correct practices can lead to faster bug detection than waiting for users to encounter them, potentially safeguarding the company's reputation.

Happy and productive testing!