"What else can I do?" Examining the Impact of Community Data on Adaptation and Quality of Reflection in an Educational Game

Title of the Paper

“What else can I do? Examining the Impact of Community Data on Adaptation and Quality of Reflection in an Educational Game”

Bibliographic Information

• Subject Area: Reflection and Adaptation in Educational Games
• Keywords: Adaptation, Reflection, Learning, Educational Games, Visualization, Community Data, Retrospective Visualization
• Authors: Erica Kleinman, Jennifer Villareale, Murtuza N. Shergadwala, Zhaoqing Teng, Andy Bryant, Jichen Zhu, Magy Seif El-Nasr
• Conference: CHI 2023 (Human Factors in Computing Systems)
• DOI: https://doi.org/10.1145/3544548.3580664

Research Background and Problem

• Identified Issues or Challenges:

  • Effective methods to support learning and promote knowledge acquisition in educational games remain unclear, particularly regarding how reflection can drive adaptive learning.
  • Retrospective visualization (visualizing players' gameplay retrospectives) is considered a potential tool for supporting reflection and adaptation, but there is a lack of empirical research in the field of educational games.
  • Comparing community data with the performance of other players might encourage behavioral adaptation; however, it could also have negative effects (e.g., players feeling inadequate or losing interest).

• Significance of the Research:

  • Research in learning sciences indicates that reflection and adaptability are critical factors in improving learning efficiency.
  • The diverse problem-solving pathways in educational games provide a unique dynamic learning environment for studying adaptability.
  • Identifying how to optimize the use of community data could have profound implications for the design and practice of educational games.

• Motivation and Related Work:

  • Reflection is considered foundational for improving learning behaviors in many learning frameworks, yet most educational games still rely primarily on written prompts.
  • While retrospective visualization with community data has been widely applied in fields like eSports, it has been less explored in educational games.
  • Previous studies suggest that community data may enhance player adaptability, but its potential negative effects and specific impacts remain unresolved.

Proposed Solution

• Proposed Approach:

  • This study examines the impact of community data comparisons on adaptability and the quality of reflection in educational games by introducing a network graph-based retrospective visualization method. This allows players to review their own and other players' gameplay trajectories during reflection.

• Innovations:

  • Unlike traditional written prompt-based reflection methods, this study explores how the combination of visualization and community data can drive adaptability.
  • It introduces retrospective visualization techniques commonly used in eSports into educational games (e.g., the parallel programming teaching game Parallel) and examines their impact on player adaptability and reflection for the first time.

• Implementation Steps and Key Techniques:

  1. Experiment Setup: 36 undergraduate computer science students participated, completing two reflection tasks under different conditions: a "self-reflection condition" based on their own two gameplay trajectories, and a "peer-reflection condition" incorporating community data.
  2. Game Tool: The educational game Parallel was used, employing a visualization metaphor to teach parallel programming concepts. The game recorded all player actions and displayed player behavior through network graphs ("playtraces").
  3. Reflection Design:
     • Self-reflection condition: Players compared their own two gameplay records.
     • Peer-reflection condition: Players compared their gameplay record with those of two other players (one similar and one different).
     • The Glyph tool was used for visualization design, with nodes representing player actions.
  4. Data Analysis:
     • Open-ended reflection responses were coded using the reflection quality framework proposed by Leijen et al.
     • McNemar-Bowker tests were used to analyze changes in willingness to adapt and reflection quality.

Research Findings

• Key Conclusions:

  • Comparing peer data significantly increased players' willingness to choose different gameplay strategies (approximately one-third of participants were willing to change strategies after reviewing peer data, p=0.004, with a large effect size).
  • Reflection quality (evaluated based on focus and levels) did not show significant changes, suggesting that community data may not directly impact reflection quality or that further research with larger samples and different contexts is needed.

• Comparison with Existing Solutions:

  • This study provides new evidence supporting the use of community data to enhance adaptability in educational games. Compared to traditional self-reflection methods, peer data more effectively stimulates players to consider behavioral changes.
  • It supplements existing knowledge on the potential negative effects of community data (e.g., feelings of inferiority) with quantitative findings and design guidelines.

• Experiment and Evaluation Results:

  • Players were significantly more likely to respond affirmatively to "whether they would adopt different strategies" under the peer-reflection condition compared to the self-reflection condition.
  • However, the distribution of reflection quality (across the three foci: technical, practical, sensitization, and four levels: description, reasoning, critique, discussion) did not show significant differences.

• Limitations and Future Directions:

  • Learning Outcomes: This study did not directly measure whether learning improved, focusing only on reflection quality and willingness to adapt. Future research should explore how community data affects long-term knowledge acquisition.
  • Individual Differences: The study did not account for personality traits (e.g., confidence or stubbornness) that might influence adaptive responses.
  • Timing: The optimal timing for exposing players to community data (e.g., during gameplay vs. post-game) remains unclear.
  • Data Presentation: Further exploration is needed on how to present community data without encouraging players to blindly follow optimal solutions.
  • Privacy and Security: Future work should address privacy protection, data fairness, and player safety when using community data.

Conclusion

This study provides the first empirical evidence of the importance of community data in enhancing player adaptability in educational games, though its standalone effects are limited. Developers are encouraged to thoughtfully integrate community data into retrospective visualizations, while researchers should further investigate the impact of data presentation methods, timing, and individual differences on learning improvements.