Software QA: Solving Issues with Combinatorial Test Design

In software testing, the design of test cases is fundamental to ensuring the reliability and robustness of apps. Combinatorial test design techniques, such as k-way testing and pairwise testing, aim to test the interactions between different input variables. However, it becomes apparent that challenges and pitfalls can hinder their effectiveness in uncovering critical errors.

This short article focuses on specific issues that could be encountered during combinatorial test design, exploring the nuances that demand careful consideration. Let’s delve into instances of incorrect input values, the importance of robust oracles, the significance of often-overlooked variable combinations, and the crucial understanding of how variables interact.

Incorrect Identification of Input Values

A k-way test set comprises all possible combinations of values for k variables. For instance, applying the Allpairs Algorithm results in a 2-way test set, encompassing all possible combinations of pairs of variable values. Consequently, a k-way fault refers to an error resulting from the interaction of values of k variables. Examining two systems, SYS1 and SYS2, revealed errors after their production releases. Both systems underwent k-way testing with k values of 2, 3, 4, and 5+.

The table displays the outcomes of k-way testing for the two systems.

Sequential checks were conducted for each k-way set. Figure 2.1 indicates that errors arising from the interaction of two or fewer input variables were 29 in SYS1 and 30 in SYS2. Errors from the interaction of three variables were 4*(30+4) in SYS2 and 12*(29+12) in SYS1. For interactions involving four variables, errors were 7*(30+4+7) in SYS2 and 1*(29+12+1) in SYS1. Errors from interactions with more than four variables were 3*(29+12+1+3) in SYS1 and 7*(30+4+7+7) in SYS2. Notably, 43 errors were not found in SYS1, and 34 errors were not found in SYS2.

In this example, the most significant errors belonged to the Not Found category. Further investigations revealed that most of the undetected errors were 1-way faults, meaning a specific value of one variable led to the error independently of other variable values. Pairwise testing should have detected these errors, but for some reason, they remained undiscovered. These Not Found errors are primarily 1-way faults that went unnoticed because certain input values were either not selected or chosen incorrectly.

The essence of the problem lies in the fact that any errors made before the application of the Allpairs Algorithm will persist. This implies that if previous test design techniques were incorrectly applied or input values were incorrect, errors in the application will persist regardless of using the Allpairs Algorithm or Orthogonal Array Testing.

Weakness in Defining Expected Results: Uncertainty in Thest Outcomes

As an example, let’s consider an application module with numerous options (something that looks like this form) and, consequently, a large number of input data combinations.

In the module, let’s say that there are 2^12 * 3 possible combinations. Here, the challenge is not in choosing incorrect input values, as all possible variable values can be exhaustively tested using the Allpairs algorithm. However, will this uncover all critical errors? Likely not, due to issues with expected results. After manipulating each combination of options in such kind of a software module, one would need to use the system for some time to verify that the options worked correctly and nothing else broke after applying the chosen options. In this case, there is no guarantee that there is no subtle, non-obvious problem easily missed.

After each test, one may need to thoroughly assess the core functions of the system. The point here is that serious defects are often not as apparent as one would like them to be, and accounting for everything in expected results becomes practically impossible.

Inattention to the Most Probable Combinations

Out of the 2^12 * 3 combinations (as we suggested in the example above), there is likely one combination of module options that will occur much more frequently than all others - the default setting. If 95% of people will never change the options from the default settings for this module, there should be good coverage of almost/near-default options. Testing all variations with a deviation from the default configuration in one option would result in a 2-digit number of tests. If testing all combinations of options with a possible deviation in two settings from the default, it would be around a hundred test cases. Testing with these test cases and all-pair test cases may yield better results than ignoring the existence of the default option. However, the Allpairs algorithm forces the tester to overlook the most probable and commonly used variable combinations.

Unknown Variable Interactions

The key factor for the success or failure of pairwise testing lies in understanding how the combination of input variables affects output data. Applying pairwise testing to two different tested applications may yield significantly different results. Some applications use fewer input data to produce output data, while others use more.

As an example, consider the program, which has three input variables (X, Y, Z) and three possible output data (1, 2, 3). The arrows indicate which variables should interact to produce a specific result. To obtain a result of 1, variables Y and X must interact; to obtain a result of 2, variables Z and X must interact. For this application, applying pairwise testing would be a suitable choice, and positive results are likely. However, in cases where there are scenarios where output data result from the interaction of more than two input variables, there is a high likelihood that pairwise testing will fail to identify errors. Simply applying pairwise testing increases the risk of overlooking critical errors in the tested application.