Playing the Imitation Game: How Perceived Generated Content Shapes Player Experience

Paper Title

Playing the Imitation Game: How Perceived Generated Content Shapes Player Experience

Publication Info

• Topic area: Player perceptions of AI- and human-generated game content and its impact on gameplay experience.
• Keywords: generative AI, procedural content generation, player experience, Turing test, perception bias, game design, human-likeness, Super Mario Bros., Sokoban, AI ethics.

Background and Problem

• Problem / challenge: Players often cannot reliably distinguish between AI- and human-generated game content, yet their perceptions of the creator significantly influence their gameplay experience. Existing studies focus on technical indistinguishability but neglect how perceptions shape player experience.
• Significance: Understanding perception biases is critical as generative AI becomes increasingly integrated into games, affecting trust, enjoyment, and the broader acceptance of AI in creative industries.
• Motivation and related work: Previous research has explored Turing-like tests in games and perception biases in AI-generated content across media. However, little is known about how players’ beliefs about content creators influence their experience, particularly in games.

Solution

• Proposed approach: A mixed-method study investigating how players perceive and experience AI- and human-generated levels in Super Mario Bros. and Sokoban.
• Novelty:
  1. Demonstrates that players’ beliefs about a level’s creator, rather than the actual creator, strongly influence their experience.
  2. Identifies subjective and fallible strategies players use to judge “human-likeness.”
  3. Reveals systematic differences in player attitudes toward procedural content generation (PCG) and generative AI.
  4. Proposes nuanced disclosure practices to address perception biases and build trust in AI-generated content.
• Procedure and key techniques:
  • Conducted a Turing-style classification task where participants judged whether levels were AI- or human-created.
  • Measured player experience using five metrics: fun, challenge, frustration, surprise, and design quality.
  • Analyzed open-ended responses to identify strategies and biases in judgments.
  • Compared attitudes toward PCG and generative AI using Likert scales and thematic analysis.

Results

• Concrete findings:
  • Players identified the true creator correctly in only 53% of trials, close to chance.
  • Levels believed to be human-made were rated higher in fun (mean = 3.72 vs. 2.92) and design quality (mean = 3.57 vs. 2.70), while levels believed to be AI-generated were rated higher in frustration (mean = 3.60 vs. 2.84).
  • Players’ beliefs about the creator had a stronger impact on ratings than the actual creator.
  • Attitudes toward generative AI were more negative than toward PCG, with concerns about reliability, ethics, and originality.
• Advantage over baselines:
  • Extends prior Turing-like tests by linking perceptions of the creator to gameplay experience, rather than focusing solely on distinguishability.
  • Highlights the role of spontaneous, unprimed judgments in shaping player experience.
• Experiments / evaluation:
  • 154 participants evaluated 60 unique levels (30 AI-generated, 30 human-created) across two games.
  • Used ordinal logistic regression and thematic analysis to analyze quantitative and qualitative data.
• Limitations and future work:
  • Limited to two 2D tile-based games (Super Mario Bros. and Sokoban), which may not generalize to other genres.
  • Relied on single-item Likert scales and brief open-ended responses, which may lack depth.
  • Observational design cannot establish causality between perception and experience.
  • Future work could explore other genres, longer play sessions, and recruit experienced game designers.

Summary

This study investigates how players’ perceptions of AI- and human-generated game levels influence their gameplay experience. Players could not reliably distinguish between AI- and human-created levels, but their beliefs about the creator significantly shaped their ratings, with human-attributed levels seen as more fun and aesthetically pleasing. Thematic analysis revealed subjective and often contradictory strategies for judging “human-likeness.” Players expressed more negative attitudes toward generative AI than procedural content generation, citing concerns about reliability, ethics, and originality. The findings highlight the need for nuanced disclosure practices to address perception biases and foster trust in AI-generated content.