Gamification Boosts Software Testing—Until the Rewards Run Out

Table of Links

• Abstract and 1 Introduction
• 2 Gamification and software testing education
  • 2.1 Software testing
  • 2.2 Gamification
  • 2.3 Gamification in software testing education
• 3 Experiment design and 3.1 Course design

  • 3.2 Participants

  • 3.3 Materials

  • 3.4 Procedure

  • 3.5 Metrics

• 4 Results and 4.1 RQ1: Student engagement
  • 4.2 RQ2: Student performance
• 5 Discussion
• 6 Threats to validity
• 7 Conclusions, Acknowledgements and References

7 Conclusions

This work presents a long gamification experience that was designed and conducted to motivate the students to create effective test suites, and makes a comparison with a non-gamified experience. The results show that the gamification benefits the improvement of both student engagement and performance. We have also observed a slight reduction of the engagement in some students toward the last stage of the experience, when they perceived that no more rewards were going to be received by keeping on working on the gamification experience. The statistical analysis indicates that the differences are significant.

In addition, the study confirms how the design of the gamification strategies is crucial for engaging students. The results obtained show that the rewards in this gamification experience acted as powerful extrinsic motivators that kept the students motivated until they perceived they were not going to get more rewards for their work. Arguably, because of the particular gamification design of this experience, the link between gamification and intrinsic motivation could not be proved. Nevertheless, the results did prove that gamification succeeds in keeping extrinsic motivation and improving performance for long periods of time. While designing a gamification experience, the professors should distribute the motivating stimulus throughout the whole experience, so that the engagement lasts until the end.

As a part of our future work, we will readjust the design of the gamification experience to include new rewards that increase student engagement in the last seminar exercises, so that we can overcome the challenge of keeping the engagement from decreasing slightly toward the last stage of the experience. In addition, we will study the effects of gamification in our software testing course during several academic years as part of a longitudinal study.

Acknowledgements

This work was supported by projects PID2019-105455GB-C32 and PID2019-105455GB-C33 funded by MCIN/AEI/10.13039/501100011033 (Spain).

References

Alhammad, M. M., & Moreno, A. M. (2018). Gamification in software engineering education: A systematic mapping. Journal of Systems and Software, 141, 131150. https://doi.org/10.1016/j.jss.2018.03.065

Andrade, S. A., De Oliveira Neves, V., & Delamaro, M. E. (2019). Software testing education: Dreams and challenges when bringing academia and industry closer together. XXXIII Brazilian Symposium on Software Engineering, SBES 2019, 47 56. https://doi.org/10.1145/3350768.3353903

Bell, J., Sheth, S., & Kaiser, G. (2011). Secret ninja testing with HALO software engineering. 4th International Workshop on Social Software Engineering, SSE 2011. https://doi.org/10.1145/2024645.2024657

Bodnar, C. A., Anastasio, D., Enszer, J. A., & Burkey, D. D. (2016). Engineers at Play: Games as Teaching Tools for Undergraduate Engineering Students. Journal of Engineering Education, 105(1), 147 200. https://doi.org/10.1002/jee.20106

Bourque, P., & Fairley, R. E. (2014). SWEBOK Guide V3.0. In IEEE Computer Society.

Buckley, I. A., & Clarke, P. J. (2018). An approach to teaching software testing supported by two different online content delivery methods. 16th LACCEI International Multi-Conference for Engineering, Education and Technology, 2018-July. https://doi.org/10.18687/LACCEI2018.1.1.377

Cacciotto, F., Fulcini, T., Coppola, R., & Ardito, L. (2021). A metric framework for the gamification of web and mobile GUI testing. IEEE 14th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2021, 126129. https://doi.org/10.1109/ICSTW52544.2021.00032

Calderón, A., Boubeta-Puig, J., & Ruiz, M. (2018). MEdit4CEP-Gam: A model-driven approach for user-friendly gamification design, monitoring and code generation in CEP-based systems. Information and Software Technology, 95, 238 264. https://doi.org/10.1016/j.infsof.2017.11.009

Capgemini. (2021). World Quality Report 2020-2021.

Clarke, P. J., Davis, D. L., Buckley, I. A., Potvin, G., Thirunarayanan, M., & Jones, E. L. (2019). An approach to integrating learning and engagement strategies (LESS) into CS class activities. ASEE Annual Conference and Exposition. https://doi.org/10.18260/1-2--32057

Clarke, P. J., Davis, D. L., Buckley, I. A., Potvin, G., Thirunarayanan, M., & Jones, E. L. (2022). Combining Learning and Engagement Strategies in a Software Testing Learning Environment. ACM Transactions on Computing Education, 22(2), 1 25. https://doi.org/10.1145/3469131

Clarke, P. J., Davis, D. L., Chang-Lau, R., & King, T. M. (2017). Impact of Using Tools in an Undergraduate Software Testing Course Supported by WReSTT. ACM Transactions on Computing Education, 17(4). https://doi.org/10.1145/3068324

Clarke, P. J., Thirunarayanan, M., Allala, S. C., Sotomayor, J. P., & Ross, M. S. (2020). Experiences of integrating learning and engagement strategies (LESs) into software engineering courses. ASEE Annual Conference and Exposition, 2020-June. https://doi.org/10.18260/1-2--34630

Clegg, B. S., Rojas, J. M., & Fraser, G. (2017). Teaching software testing concepts using a mutation testing game. 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering and Education Track, ICSE-SEET 2017, 33 36. https://doi.org/10.1109/ICSESEET.2017.1

Costa, I. E. F., & Oliveira, S. R. B. (2019). A systematic strategy to teaching of exploratory testing using gamification. 14th International Conference on Evaluation of Novel Approaches to Software Engineering, ENASE 2019, 307 314. https://doi.org/10.5220/0007711603070314

Costa, I. E. F., & Oliveira, S. R. B. (2020). The use of gamification to support the teaching-learning of software exploratory testing: An experience report based on the application of a framework. IEEE Frontiers in Education Conference, FIE 2020, 2020-October, 19. https://doi.org/10.1109/FIE44824.2020.9273943

Dal Sasso, T., Mocci, A., Lanza, M., & Mastrodicasa, E. (2017). How to gamify software engineering. 24th IEEE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2017, 261 271. https://doi.org/10.1109/SANER.2017.7884627

Deak, A., Stålhane, T., & Sindre, G. (2016). Challenges and strategies for motivating software testing personnel. Information and Software Technology, 73, 115. https://doi.org/10.1016/j.infsof.2016.01.002

Deterding, S., O’Hara, K., Sicat. M., Dixon., & Nacker, L (2021). Gamification: Using game design elements in non-gaming contexts. Extended Abstracts on Human Factors in Computing Systems, CHI EA 2011, 2425 2428. https://doi.org/10.1145/1979742.1979575

Dichev, C., & Dicheva, D. (2017). Gamifying education: what is known, what is believed and what remains uncertain: a critical review. International Journal of Educational Technology in Higher Education, 14. https://doi.org/10.1186/s41239-017-0042-5

Elbaum, S., Person, S., Dokulil, J., & Jorde, M. (2007). Bug hunt: Making early software testing lessons engaging and affordable. 29th International Conference on Software Engineering, ICSE 2007, 688 697. https://doi.org/10.1109/ICSE.2007.23

Elgrably, I. S., & Oliveira, S. R. B. (2018). Gamification and evaluation of the use the agile tests in software quality subjects: The application of case studies. 13th International Conference on Evaluation of Novel Approaches to Software Engineering, ENASE 2018, 2018-March, 416423. https://doi.org/10.5220/0006800304160423

Elgrably, I. S., & Oliveira, S. R. B. (2022). Using flipped classroom to promote active learning and engagement in a Software Testing subject remotely during the COVID-19 pandemic. IEEE Frontiers in Education Conference, FIE 2022, 16. https://doi.org/10.1109/FIE56618.2022.9962379

Fraser, G. (2017). Gamification of Software Testing. 12th IEEE/ACM International Workshop on Automation of Software Testing, AST 2017, 2 7. https://doi.org/10.1109/AST.2017.20

Fraser, G., Gambi, A., Kreis, M., & Rojas, J. M. (2019). Gamifying a Software Testing Course with Code Defenders. 50th ACM Technical Symposium on Computer Science Education, SIGCSE 2019, 571 577. https://doi.org/10.1145/3287324.3287471

Fraser, G., Gambi, A., & Rojas, J. M. (2020). Teaching Software Testing with the Code Defenders Testing Game:Experiences and Improvements. 13th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2020, 461464. https://doi.org/10.1109/ICSTW50294.2020.00082

Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59109. https://doi.org/10.3102/00346543074001059

Fu, Y., & Clarke, P. J. (2016). Gamification-based cyber-enabled learning environment of software testing. ASEE Annual Conference and Exposition, 2016-June. https://doi.org/10.18260/p.27000

Garaccione, G., Fulcini, T., & Torchiano, M. (2022). GERRY: a gamified browser tool for GUI testing. 1st International Workshop on Gamification of Software Development, Verification and Validation, Gamify 2022, 2 9. https://doi.org/10.1145/3548771.3561408

Garcia, I., Pacheco, C., Méndez, F., & Calvo-Manzano, J. A. (2020). The effects of game-based learning in the acquisition of “soft skills“on undergraduate software engineering courses: A systematic literature review. Computer Applications in Engineering Education, 28(5), 13271354. https://doi.org/10.1002/cae.22304

Garousi, V., Rainer, A., Lauvås, P., & Arcuri, A. (2020). Software-testing education: A systematic literature mapping. Journal of Systems and Software, 165. https://doi.org/10.1016/J.JSS.2020.110570

Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamification work? - A literature review of empirical studies on gamification. 47th Hawaii International Conference on System Sciences, 30253034. https://doi.org/10.1109/HICSS.2014.377

International Software Testing Qualification Board (ISTQB). (2020). ISTQB Standard glossary of terms used in Software Testing, version 3.5. https://glossary.istqb.org/

Isomöttönen, V., & Lappalainen, V. (2012). CS1 with games and an emphasis on TDD and unit testing: Piling a trend upon a trend. ACM Inroads, 3(3), 62 68. https://doi.org/10.1145/2339055.2339073

Jesus, G. M. de, Ferrari, F. C., Paschoal, L. N., Souza, S. D. R. S. de, Porto, D. D. P., & Durelli, V. H. S. (2020). Is It Worth Using Gamification on Software Testing Education? An Extended Experience Report in the Context of Undergraduate Students. Journal of Software Engineering Research and Development, 8, 6:1-6:19. https://doi.org/10.5753/jserd.2020.738

Jesus, G. M. de, Ferrari, F. C., Porto, D. de P., & Fabbri, S. C. P. F. (2018). Gamification in software testing: A characterization study. III Brazilian Symposium on Systematic and Automated Software Testing, SAST 2018, 39 48. https://doi.org/10.1145/3266003.3266007

Khaleel, F. L., Ashaari, N. S., Wook, T. S. M. T. W. T., & Ismail, A. (2016). The Architecture of Dynamic Gamification Elements Based Learning Content. Journal of Convergence Information Technology, 11(3), 164 177.

Khandelwal, S., Sripada, S. K., & Raghu Reddy, Y. (2017). Impact of gamification on code review process - An experimental study. 10th Innovations in Software Engineering Conference, ISEC 2017, 122 126. https://doi.org/10.1145/3021460.3021474

Kosa, M., Yilmaz, M., O'Connor, R. V., & Clarke, P. M. (2016). Software engineering education and games: A systematic literature review. Journal of Universal Computer Science, 22(12), 1558– 1574. https://doi.org/10.3217/jucs-022-12-1558

Krasner, H. (2021). The Cost of Poor Software Quality in the US: A 2020 Report.

Lőrincz, B., Iudean, B., & Vescan, A. (2021). Experience report on teaching testing through gamification. 3rd International Workshop on Education through Advanced Software Engineering and Artificial Intelligence, EASEAI 2021, 15-22. https://doi.org/10.1145/3472673.3473960

Marabesi, M., & Silveira, I. F. (2019). Towards a gamified tool to improve unit test teaching. 14th Latin American Conference on Learning Technologies, LACLO 2019, 1218. https://doi.org/10.1109/LACLO49268.2019.00013

Micari, M., & Pazos, P. (2012). Connecting to the Professor: Impact of the Student Faculty Relationship in a Highly Challenging Course. College Teaching, 60(2), 4147. https://doi.org/10.1080/87567555.2011.627576

Milosz, M., & Milosz, E. (2020). Gamification in engineering education - A preliminary literature review. IEEE Global Engineering Education Conference, EDUCON 2020, 2020-April. https://doi.org/10.1109/EDUCON45650.2020.9125108

Myers, G. J., Badgett, T., & Sandler, C. (2012). The art of software testing (3rd Edition). Wiley.

Papadakis, M., Kintis, M., Zhang, J., Jia, Y., Traon, Y. Le, & Harman, M. (2019). Mutation Testing Advances: An Analysis and Survey. Advances in Computers, 112, 275378. https://doi.org/10.1016/bs.adcom.2018.03.015

Pedreira, O., García, F., Brisaboa, N., & Piattini, M. (2015). Gamification in software engineering - A systematic mapping. Information and Software Technology, 57, 157168. https://doi.org/10.1016/j.infsof.2014.08.007

Peixoto, M., & Silva, C. (2017). A gamification requirements catalog for educational software: Results from a systematic literature review and a survey with experts. ACM Symposium on Applied Computing, Part F128005, 1108 1113. https://doi.org/10.1145/3019612.3019752

Rojas, J. M., & Fraser, G. (2016). Code Defenders: A Mutation Testing Game. 9th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2016, 162167. https://doi.org/10.1109/ICSTW.2016.43

Ruiperez-Valiente, J. A., Gomez, M. J., Martinez, P. A., & Kim, Y. J. (2021). Ideating and Developing a Visualization Dashboard to Support Teachers Using Educational Games in the Classroom. IEEE Access, 9, 83467 83481. https://doi.org/10.1109/ACCESS.2021.3086703

Sardi, L., Idri, A., & Fernández-Alemán, J. L. (2017). A systematic review of gamification in e-Health. Journal of Biomedical Informatics, 71, 31 48. https://doi.org/10.1016/j.jbi.2017.05.011

Sherif, E., Liu, A., Nguyen, B., Lerner, S., & Griswold, W. G. (2020). Gamification to Aid the Learning of Test Coverage Concepts. 32nd IEEE International Conference on Software Engineering Education and Training, CSEE&T 2020, 320324. https://doi.org/10.1109/CSEET49119.2020.9206224

Sheth, S., Bell, J., & Kaiser, G. (2015). A Gameful approach to teaching software design and software testing. In Computer Games and Software Engineering (pp. 91113). https://doi.org/10.1201/b18453

Sheth, S., Bell, J., & Kaiser, G. (2013). A competitive-collaborative approach for introducing software engineering in a CS2 class. 26th International Conference on Software Engineering Education and Training, CSEE&T 2013, 41 50. https://doi.org/10.1109/CSEET.2013.6595235

Silvis-Cividjian, N. (2021). Awesome Bug Manifesto: Teaching an Engaging and Inspiring Course on Software Testing (Position Paper). 3rd International Workshop on Software Engineering Education for the Next Generation, SEENG 2021, 1620. https://doi.org/10.1109/SEENG53126.2021.00010

Souza, M. R. d. A., Veado, L., Moreira, R. T., Figueiredo, E., & Costa, H. (2018). A systematic mapping study on game-related methods for software engineering education. Information and Software Technology, 95, 201 218. https://doi.org/10.1016/j.infsof.2017.09.014

Subhash, S., & Cudney, E. A. (2018). Gamified learning in higher education: A systematic review of the literature. Computers in Human Behavior, 87, 192206. https://doi.org/10.1016/j.chb.2018.05.028

Trinidad, M., Calderon, A., & Ruiz, M. (2021). GoRace: A Multi-Context and Narrative-Based Gamification Suite to Overcome Gamification Technological Challenges. IEEE Access, 9, 6588265905. https://doi.org/10.1109/ACCESS.2021.3076291

Valle, P. H. D., Toda, A. M., Barbosa, E. F., & Maldonado, J. C. (2017). Educational games: A contribution to software testing education. IEEE Frontiers in Education Conference, FIE 2017, 2017-October, 1 8. https://doi.org/10.1109/FIE.2017.8190470

Vos, T. E. J., Fraser, G., Martinez-Ortiz, I., Prada, R., Silva, A. R., & Prasetya, I. S. W. B. (2020). Tutorial on a Gamification Toolset for Improving Engagement of Studentsin Software Engineering Courses. In M. Daun, E. Hochmuller, S. Krusche, B. Brugge, & B. Tenbergen (Eds.), 32nd IEEE Conference on Software Engineering Education and Training, CSEE&T 2020 (pp. 291 293). IEEE.

Werbach, K., & Hunter, D. (2012). For the Win: How Game Thinking Can Revolutionize Your Business. Wharton School Press.

Zivkovic, T., & Zivkovic, M. (2021). Survey of Learning Environments for Software Testing Education. 7th Conference on the Engineering of Computer Based Systems, ECBS 2021, 19. https://doi.org/10.1145/3459960.3459971