Mining Player Experience Trends From Game Reviews Using Large Language Models

Paper Title

Mining Player Experience Trends From Game Reviews Using Large Language Models

Publication Info

• Topic area: Analyzing player experience trends in video games using computational methods.
• Keywords: Player experience, game reviews, large language models, semantic similarity, emotional challenge, nostalgia, meaning, boredom, longitudinal trends.

Background and Problem

• Problem / challenge: Existing methods for studying player experiences rely on questionnaires, which are limited by sample size and lack longitudinal data. While game reviews offer a vast data source, prior methods for analyzing them have been rudimentary or manual, limiting insights into player experiences.
• Significance: Understanding player experience trends can inform game design, improve player satisfaction, and highlight the evolution of games as an art form.
• Motivation and related work: Prior work has used sentiment analysis and topic modeling on game reviews but lacked the depth to map reviews to established player experience constructs. Recent advances in LLMs offer new opportunities to bridge this gap by predicting questionnaire responses from free-form text.

Solution

• Proposed approach: A novel method to map game reviews to player experience questionnaire items using LLM-based text embeddings, enabling scalable and longitudinal analysis of player experiences.
• Novelty:
  1. First large-scale analysis of game reviews mapped to player experience constructs using LLM embeddings.
  2. Identification of longitudinal trends in player experiences (e.g., emotional challenge, nostalgia).
  3. Introduction of a scalable method for discovering reference games for specific experiences.
  4. Correlation analysis linking player experience constructs to review scores.
• Procedure and key techniques:
  • Use OpenAI’s text-embedding-3-large model to compute semantic similarities between reviews and questionnaire items.
  • Apply thresholding to identify reviews strongly agreeing with specific experience constructs.
  • Analyze trends using percentages of reviews above thresholds, breakdowns by genres and games, and qualitative content analysis.
  • Correlate review scores with experience constructs to identify priorities for game design.

Results

• Concrete findings:
  • Increasing trends in emotional challenge, meaning, nostalgia, and audiovisual appeal.
  • Rising boredom in reviews, often attributed to bad writing, repetitive gameplay, and poor sequels.
  • Correlations between review scores and constructs such as joy (r = 0.53), mastery (r = 0.40), and audiovisual appeal (r = 0.39).
• Advantage over baselines: The embedding-based method is more scalable and detailed than prior manual or rudimentary NLP approaches, enabling longitudinal and multidimensional analysis of player experiences.
• Experiments / evaluation:
  • Dataset: 152,143 user reviews from Metacritic (2010–2024).
  • Metrics: Semantic similarity, percentage of reviews above thresholds, Pearson correlations.
  • Validation: Human-coded review-item agreement showed moderate-to-strong correlation with embedding-based similarity.
• Limitations and future work:
  • Trends reflect review discourse rather than direct player experiences.
  • Variability in review quality and brevity may introduce noise.
  • Future work could include expert reviews, other questionnaires, and social media data, as well as improved validation with ground truth datasets.

Summary

This study introduces a scalable method for analyzing player experience trends in game reviews by mapping them to questionnaire constructs using LLM embeddings. Key findings include rising trends in emotional challenge, meaning, and nostalgia, alongside increasing boredom. Correlation analysis highlights which experiences (e.g., joy, mastery) are most associated with high review scores, offering actionable insights for game developers. The method also enables the discovery of reference games for specific experiences. While limitations exist, the approach provides a foundation for future research into longitudinal and multidimensional player experience analysis.