Gathering Self-Report Data in Games through NPC Dialogues: Effects on Data Quality, Data Quantity, Player Experience, and Information Intimacy

Title of the Paper

Collecting Self-Report Data in Games through NPC Dialogues: Effects on Data Quality, Quantity, Player Experience, and Information Intimacy

Paper Information

• Subject Area: Human-Computer Interaction, Game User Research, Self-Report Data Collection
• Keywords: Self-report, Questionnaire, Non-Player Characters (NPCs), Player Experience, Intimacy, Game User Research (GUR), Survey

Research Background and Issues

• Problems and Challenges: Collecting self-report data in games is critical for design decisions and user experience. However, the impact of embedded questionnaire designs (e.g., NPC dialogues vs. traditional questionnaires) remains unclear, particularly regarding data quality, quantity, and potential effects on player privacy.
• Significance of the Research: Designing embedded self-report collection tools can reduce interruptions to player experience, enhance data usability, and improve player engagement. However, without sufficient research, such designs may unnecessarily harm user experience or compromise data collection effectiveness.
• Motivation and Related Work:
  • Previous studies suggest that embedded questionnaire formats (e.g., through NPC dialogues) may be more suitable than traditional methods, but there is a lack of quantitative evaluation of their specific impacts in real gaming contexts.
  • While there is theoretical support for NPC dialogues as an ecologically valid questionnaire format, their effects on data quality and player behavior have not been fully explored.

Proposed Solution

• Research Method: Conduct a user study to compare the effectiveness of data collection through NPC dialogues versus traditional overlay questionnaires. The study involved 166 participants in a gaming experiment to evaluate the performance of the two data collection methods across multiple dimensions.
• Innovations: A detailed investigation into the effects of NPC dialogue formats on data quality, data quantity, player experience, and perceptions of information intimacy, with a comprehensive evaluation of their implications for future game design.
• Implementation Steps:
  1. Designed and developed a 3D platform game featuring an NPC called "Assist Cube," which provided guidance and facilitated self-report data collection tasks.
  2. Tested under two experimental conditions:
     • NPC-integrated Questionnaire (NPQ) condition: Embedded questionnaires and response options presented through NPC dialogues.
     • Traditional Overlay Questionnaire (TOQ) condition: Standard game interface used to collect questionnaire data.
  3. Data was evaluated using various methods: the number of responses to 5-point scale items, word count for open-ended responses, data quality and player experience ratings (e.g., autonomy, immersion, meaningfulness), and perceptions of information intimacy.

Research Findings

• Specific Results:
  • Data Quality: NPC dialogues and traditional questionnaires showed comparable accuracy and quality, consistent with post-game questionnaires.
  • Data Quantity: NPC dialogues significantly increased the number of responses to 5-point scale items, though there was no significant difference in word count for open-ended responses.
  • Information Intimacy:
    • No significant difference in the objective intimacy of shared information between the two conditions.
    • However, players perceived the information shared through NPC dialogues as more intimate (e.g., more private and personal), which could be viewed as more intrusive.
  • Player Experience:
    • No significant differences were observed across multiple experience metrics (autonomy, immersion, curiosity, sense of control) between the two conditions.
    • However, NPC dialogues significantly enhanced the "meaningfulness" rating of the game.
  • Potential Mechanism: A strong positive correlation was found between player-NPC homophily (shared values, beliefs, etc.) and the enhanced sense of meaningfulness, suggesting that fostering player resonance can improve the overall experience.
• Advantages:
  • NPC dialogues provide a more natural interaction format, particularly excelling in collecting quantitative data (e.g., 5-point scale questions).
  • They can enhance players' emotional and meaningful engagement with the game.
• Limitations:
  • The increased perception of information intimacy may lead to feelings of privacy intrusion, potentially reducing player acceptance of data collection.
  • Limited impact on increasing the quantity of data for open-ended questions.
• Future Directions:
  • Explore diverse dialogue designs to accommodate different game types and player needs.
  • Validate the long-term effects of this method in real-world gaming environments (outside laboratory settings).
  • Investigate the causal relationship between player-NPC homophily and shared information.