Vibe Check: Understanding the Effects of LLM-Based Conversational Agents' Personality and Alignment on User Perceptions in Goal-Oriented Tasks

Paper Title

Vibe Check: Understanding the Effects of LLM-Based Conversational Agents' Personality and Alignment on User Perceptions in Goal-Oriented Tasks

Publication Info

• Topic area: Human-computer interaction, focusing on personality design in conversational agents.
• Keywords: Large language models, conversational agents, personality alignment, user perceptions, goal-oriented tasks, Big Five traits, adaptive calibration, human-AI interaction, personality prompting, user experience.

Background and Problem

• Problem / challenge: Prior research on conversational agent (CA) personality has focused on binary trait manipulations or extreme personality expressions, leaving gaps in understanding moderate personality levels and their effects on user perceptions. Additionally, personality alignment has been treated as binary rather than continuous, limiting insights into nuanced compatibility.
• Significance: As LLM-based CAs become integral to domains like education, healthcare, and professional practice, understanding how personality expression and alignment influence user perceptions is critical for optimizing trust, adoption, and usability.
• Motivation and related work: Early studies demonstrated that users perceive personality in CAs and prefer agents that mirror their own traits. However, mixed findings on alignment effects and limited exploration of medium personality levels have left unanswered questions about optimal personality design and its role in enhancing user experience.

Solution

• Proposed approach: Trait Modulation Keys (TMK), a modular prompting framework enabling systematic control of Big Five personality traits at low, medium, and high expression levels without relying on persona-based backstories.
• Novelty:
  1. Introduced TMK for concurrent control across all Big Five traits at multiple expression levels, including medium-level expression.
  2. Demonstrated an inverted-U relationship between personality expression intensity and user perceptions, with medium expression outperforming extremes.
  3. Established personality alignment as a key design factor, highlighting trait-specific effects and emergent user compatibility profiles.
  4. Provided empirical evidence linking personality expression and alignment to user perceptions in goal-oriented tasks.
• Procedure and key techniques:
  • Conducted a 3×1 between-subjects study (N=150) using a travel-planning CA with three personality profiles (Low, Medium, High).
  • Measured user perceptions across six dimensions (Intelligence, Enjoyment, Anthropomorphism, Intention to Adopt, Trust, Likeability) and calculated personality alignment scores using normalized Euclidean distance.
  • Validated TMK’s control fidelity across 243 trait-level configurations using psychometric inventories.

Results

• Concrete findings:
  • Medium CA consistently produced higher user perceptions across all six measures compared to Low CA and outperformed High CA on Intelligence and Likeability.
  • Personality alignment positively correlated with all six measures, with Extraversion and Emotional Stability showing the strongest effects.
  • Individual trait mismatches (e.g., Conscientiousness, Extraversion) negatively impacted user perceptions, while Openness had minimal influence.
  • Cluster analysis identified three user groups based on alignment patterns: Well-Aligned, Extraversion-Misaligned, and Globally-Misaligned.
• Advantage over baselines:
  • Medium CA outperformed Low CA across all measures and High CA on Intelligence and Likeability, revealing an inverted-U relationship in personality expression effects.
  • TMK achieved 92.3% accuracy in shaping targeted personality profiles, enabling nuanced control beyond binary manipulations.
• Experiments / evaluation:
  • Participants interacted with a text-based CA for a 10-minute NYC trip-planning task, completing subtasks and rating perceptions post-interaction.
  • Validation tests confirmed TMK’s reliability in producing distinct personality profiles across low, medium, and high levels.
• Limitations and future work:
  • Limited generalizability due to sample demographics (US adults, native English speakers).
  • Focused on travel-planning tasks; effects in other domains remain unexplored.
  • Interaction duration was short, limiting insights into long-term user perceptions.
  • Medium personality levels were harder to control precisely, suggesting room for refinement in prompting frameworks.

Summary

This study developed the Trait Modulation Keys (TMK) framework to systematically control personality expression in LLM-based conversational agents, revealing that medium personality levels optimize user perceptions across Intelligence, Enjoyment, Anthropomorphism, Intention to Adopt, Trust, and Likeability. Personality alignment further enhanced outcomes, with Extraversion and Emotional Stability emerging as key traits. Findings highlight the importance of adaptive calibration and trait-specific prioritization in CA design, offering actionable insights for creating personality-aware systems that balance usability and ethical considerations. Future research should explore personality effects across diverse domains, cultures, and extended interactions.