Cracking the Case Together: Role Perceptions in Human-AI Mystery Solving Dialogues

Paper Title

Cracking the Case Together: Role Perceptions in Human-AI Mystery Solving Dialogues

Publication Info

• Topic area: Human-AI collaboration and conversational dynamics.
• Keywords: Human-AI interaction, role perception, collaboration, conversational AI, team dynamics, emotion analysis, mystery solving, Large Language Models, trust, team cohesion.

Background and Problem

• Problem / challenge: While AI systems increasingly participate in collaborative tasks, the role humans ascribe to AI (e.g., as a tool or teammate) and its impact on interaction dynamics and outcomes remain underexplored.
• Significance: Understanding role perceptions can inform the design of AI systems that foster effective and positive human-AI collaboration without compromising task performance.
• Motivation and related work: Prior studies have shown that AI's conversational capabilities can lead to anthropomorphic perceptions, influencing trust and collaboration. However, gaps remain in understanding how these perceptions shape interaction style, emotional tone, and task outcomes.

Solution

• Proposed approach: An exploratory user study analyzing human-AI collaboration during a mystery-solving task, using Anthropic’s Claude 3.5 Sonnet v2 model as the AI partner.
• Novelty:
  1. Empirical investigation of how role perceptions (tool vs. teammate) affect collaboration dynamics and emotions.
  2. Integration of self-reports, task performance metrics, and LLM-based emotion coding for a comprehensive analysis.
  3. Insights into the emotional and interactional dimensions of human-AI teaming.
• Procedure and key techniques:
  • Participants (n=67) collaborated with the AI to solve a mystery game using a textual chat interface.
  • Data collected included self-reports on role perception, team cohesion, and interaction style, as well as task performance metrics and chat logs.
  • Emotional tone of conversations analyzed using Plutchik’s taxonomy and LLM-based semantic coding.

Results

• Concrete findings:
  • Participants who perceived the AI as a teammate reported higher team cohesion (M=4.03 vs. 3.63, p=.02) and more collaborative interaction styles (M=3.56 vs. 2.37, p<.01).
  • Emotional analysis showed higher expressions of joy (M=1.14 vs. 0.62, p<.01) and trust (M=1.67 vs. 1.23, p=.02) in teammate-like interactions.
  • Task performance (M=3.28 vs. 2.87, p=.16) and conversational length (words: M=4,804 vs. 3,594, p=.14) did not significantly differ between groups.
• Advantage over baselines:
  • Teammate-like perceptions correlated with more positive emotions and collaborative dialogue, though task performance remained unaffected.
• Experiments / evaluation:
  • Mystery-solving task with distributed clues required active collaboration.
  • Metrics included task accuracy, number of clues exchanged, and emotional tone (joy, trust, fear, surprise, anger).
• Limitations and future work:
  • Small sample size limited detection of smaller effects.
  • Short study duration may not reflect long-term human-AI interactions.
  • Emotion analysis reliability varied across LLMs; further validation needed.
  • Future studies should explore longitudinal interactions and richer communication media.

Summary

This study investigated how humans perceive AI as a tool or teammate during collaborative mystery solving. Results showed that teammate-like perceptions enhanced team cohesion, collaborative interaction, and positive emotions (e.g., joy, trust), though task performance was unaffected. These findings suggest that designing AI systems to foster positive emotional engagement and clear role expectations can improve human-AI collaboration. Future research should explore long-term dynamics, larger samples, and the impact of richer interaction modalities.