Trading Accuracy for Enjoyment? Data Quality and Player Experience in Data Collection Games

Title of the Paper

Trading Accuracy for Enjoyment? Data Quality and Player Experience in Data Collection Games

Paper Information

• Research Domain: Study on data quality and gamified user experience
• Keywords: data collection games, gamification, data quality, human-subject research, player experience, experimental design, validity threats, linguistic experiments

Research Background and Problem

• Identified Problem or Challenge: Traditional surveys or experiments are often tedious, leading to poor participant retention and data quality issues. Data collection games have been proposed as a solution to enhance participant engagement while obtaining data as reliable as traditional methods. However, this assumption lacks empirical validation.
• Significance: The reliability of data collection directly impacts the quality of human-subject research and the credibility of research outcomes. Gamification, as a potential means to boost motivation and improve user experience, holds significant value, but it remains unclear whether it can enhance engagement while maintaining data quality.
• Research Motivation and Related Work:
  • Gamification has been applied in fields such as citizen science, but the focus has mostly been on increasing data volume, with limited validation of data quality.
  • Current studies on data collection games lack randomized controlled experiments, and existing findings are inconsistent.
  • The authors propose a specific design framework (Intrinsic Elicitation) aimed at reducing validity threats in games and provide experimental data to validate the framework's effectiveness.

Solution

• Proposed Method or Solution:

  • Developed a data collection game called Adjective Game to gather human linguistic data (specifically, patterns of adjective order usage).
  • Designed a comparative experiment to evaluate participant experience and data quality under game conditions versus standard linguistic experiment conditions.
  • Defined accuracy using two operationalizations: the first based on English grammar rules, and the second based on participants' subjective judgments.

• Innovations:

  • Employed a unique design framework (Intrinsic Elicitation) to mitigate validity threats.
  • Conducted the first rigorous experimental validation of the hypothesis that data collection games can enhance participant enjoyment while maintaining data quality.

• Implementation Steps and Key Techniques:

  1. Game Design:
     • Core mechanics involved players selecting three words (two adjectives and one noun) to describe shapes on the screen, thereby clearing blocks and collecting linguistic data.
     • Followed the three principles of Intrinsic Elicitation: necessity, centrality, and authenticity.
  2. Experimental Design:
     • Established two experimental conditions: data collection game and traditional experimental task.
     • Conducted two experiments to compare participants' self-reported enjoyment and data accuracy.
  3. Data Analysis:
     • Used statistical methods (e.g., Mann-Whitney U test) to analyze differences in participant experience and data accuracy between the two conditions.

Research Findings

• Specific Findings:

  • The game condition significantly enhanced participants' interest and experience (with effect sizes of d = .70 and .73 in the current study, indicating medium to large effects).
  • Data collected via the game was significantly lower in quality compared to traditional experimental tasks, though still more accurate than random responses (with effect sizes of d = -.68 and -.40 in the current study).
  • In terms of time efficiency, the game task required more time per input on average.

• Advantages Compared to Existing Solutions:

  • Data collection games can attract participants, alleviating issues of dropout and item omission caused by boredom in traditional experiments.
  • The game integrates linguistic data collection mechanisms and includes optimizations such as randomization of participant input data.

• Experimental or Evaluation Results:

  • Participants' enjoyment under the game condition was significantly higher than in the control experimental condition, supporting the claim that gamification improves experience.
  • However, data quality was affected by validity threats, revealing a notable trade-off between enjoyment and quality in data collection games.

• Limitations and Future Directions:

  • Limitations:
    • While the Intrinsic Elicitation framework effectively mitigates some validity threats, it cannot completely eliminate them.
    • The cognitive load associated with game mechanics may lead to inaccurate behavioral responses.
  • Future Directions:
    • Test the generalizability of results with other types of data and different game designs.
    • Investigate the specific mechanisms by which the framework influences participant behavior and data quality, particularly under "experimental task" versus "game task" contexts.
    • Explore design improvements to decouple high enjoyment from high data quality and identify specific factors contributing to both.

Through this study, the authors provide important insights into data collection games, emphasizing the need to carefully balance user experience and data quality when adopting gamification. They also call for further experimental research to establish more reliable design methods and gamification implementation strategies.