Hybrid Search Strategy for Gamification Research in Education

Table of Links

Abstract and 1 Introduction

2. Background and Related Work and 2.1. Gamification

2.2. Game Design Elements and 2.3. Gamification Effects

2.4. Related Work on Gamification Negative Effects

3. Systematic Mapping and 3.1. The Research Questions

3.2. Search Strategy and 3.3. Inclusion and Exclusion Criteria

3.4. Applying the Search Strategy

3.5. Data Extraction

4. Systematic Mapping Results

5. Focus Group: Developer Perception on the Negative Effects of Game Design Elements

5.1. Context and Participant Characterization

5.2. Focus Group Design

5.3. The Developers’ Perception on The Negative Effects

5.4. On the Perceived Usefulness, Ease of use and Intent of Adoption of Mapped Negative Effects

5.5. Participant Feedback

6. Limitations

7. Concluding Remarks

7.1. Future Research Directions

Acknowledgements and References

3.2. Search Strategy

We decided to use a hybrid search strategy, combining a database search on Scopus with forward and backward snowballing [15]. Hybrid strategies were found to be capable of achieving an appropriate balance of precision and recall when looking for primary studies [15]. To design the search string for the database search on Scopus, we used the PICOC (Population, Intervention, Comparison, Outcome, Context) criteria [48] as follows:

• Population: gamification software.

• Intervention: game design elements.

• Comparison: none.

• Outcomes: negative effects.

• Context: education/learning.

After that, we extracted the basic terms from the PICO criteria (gamification, education/learning, negative effects) and added synonyms and related terms. We decided not to include the intervention’s terms, as we conducted the database search based on title, keywords, and abstract, where details on game design elements could have been omitted.

We added the following synonyms and related terms:

• Gamification: gamify, gamified, gamifying.

• Education/learning: information, teaching, curriculum, pedagogy, didactics, training, instruction.

• Negative: damaging, prejudicious, prejudicial, detrimental, counterproductive, inappropriate, harmful, perilous, limiting.

Finally, we applied AND and OR logic operators to connect the terms, resulting in the following search string:

(gamification OR gamify OR gamified OR gamifying) AND (education OR learning OR information OR teaching OR curriculum OR pedagogy OR didactics OR training OR instruction) AND (negative OR damaging OR prejudicious OR prejudicial OR detrimental OR counterproductive OR inappropriate OR harmful OR perilous OR limiting)

As snowballing support tool, we used Publish or Perish [49], a software program that allows retrieving academic citations using information from Scopus and Google Scholar.

3.3. Inclusion and Exclusion Criteria

The inclusion and exclusion criteria can be found in Table 4 and 5, respectively. The exclusion criteria were derived from the inclusion criteria and provide details on our three-phase filtering procedure. To organize and filter the documents throughout the systematic mapping, we used Rayyan, a free web application to support systematic review authors [50].