Toward Natural and Companionable Virtual Agents via Cross-Temporal Emotional Modeling

Paper Title

Toward Natural and Companionable Virtual Agents via Cross-Temporal Emotional Modeling

Publication Info

• Topic area: Long-term emotional and behavioral modeling in virtual companion agents.
• Keywords: Cross-temporal modeling, emotional state, companion agents, coherence, harmony, adaptive interaction, behavioral dynamics, virtual agents, user experience, human–AI interaction.

Background and Problem

• Problem / challenge: Existing virtual agents lack the ability to model long-term emotional and behavioral dynamics, resulting in interactions that feel episodic and inauthentic. Current systems often treat behavior and affect in isolation, failing to integrate past interactions into evolving emotional states that influence future behaviors.
• Significance: Addressing this limitation is critical for creating agents that can sustain coherent and emotionally engaging relationships over time, with applications in education, healthcare, and social interaction.
• Motivation and related work: Prior work has explored memory, personality modeling, and emotional expression in agents, but these approaches often focus on short-term engagement or static emotional profiles. Few systems model the bidirectional dynamics between long-term affective accumulation and moment-to-moment emotional expression, leaving a gap in achieving temporal coherence and emotional harmony.

Solution

• Proposed approach: Cross-Temporal Emotional Modeling (CTEM), a framework that integrates long-term behavioral history with real-time emotional expression to enable coherent and harmonious interactions.
• Novelty:
  1. Introduction of CTEM, a computational framework coupling long-term behavioral accumulation with dynamic emotional states.
  2. Development of Auri, a lightweight companion agent implementing CTEM on an instant messaging platform.
  3. Empirical evaluation of CTEM’s impact on perceived coherence and harmony through a 21-day user study.
• Procedure and key techniques:
  • CTEM formalizes emotional states as a tuple of physio-emotional state, motivational drives, memory, and personality.
  • Behaviors are generated and integrated into a structured inventory, balancing past experiences, current conditions, and future intentions.
  • Adaptive interaction and feedback-driven adaptation ensure real-time modulation of emotional states and long-term coherence.
  • Safeguards, such as safety constraints and ethical design principles, mitigate risks of over-dependence and ensure user well-being.

Results

• Concrete findings:
  • CTEM improved perceived coherence and harmony, with participants reporting higher emotional resonance and alignment.
  • Emotional improvement ratings increased significantly from baseline to CTEM-enabled phases, particularly in groups where emotional modules were restored.
  • Usage experience ratings showed significant improvement, with the largest gains observed when transitioning from baseline to CTEM.
• Advantage over baselines:
  • CTEM outperformed a baseline system without cross-temporal modeling in terms of perceived coherence, emotional harmony, and overall user experience.
  • Modules for adaptive interaction (AdI) and emotional state updating (ESU) contributed more to coherence and harmony than behavior generation and integration (BGI).
• Experiments / evaluation:
  • A 21-day mixed-methods study with 96 participants aged 18–26, including a main study with ablation conditions and a sub-study comparing CTEM to a baseline system.
  • Data collected through surveys, interviews, and computational logs, with analysis using statistical models and thematic coding.
• Limitations and future work:
  • Limited study duration and demographic focus (18–26 years old) restrict generalizability.
  • Future work should include longer-term deployments, more diverse populations, and additional modalities (e.g., voice).
  • Ethical concerns such as over-dependence and privacy risks require further exploration.

Summary

This paper introduces Cross-Temporal Emotional Modeling (CTEM), a framework that integrates long-term behavioral histories with dynamic emotional states to enhance the coherence and harmony of virtual companion agents. Implemented in Auri, a lightweight agent deployed on an instant messaging platform, CTEM was evaluated in a 21-day study with 96 participants. Results demonstrated significant improvements in perceived coherence, emotional harmony, and user experience compared to a baseline system. The study also identified key design tensions, such as stability versus variation and coherence versus flexibility, providing actionable insights for future companion agent systems. While promising, further research is needed to address ethical considerations and extend findings to broader contexts.