Breakdowns in Conversational AI: Interactional Failures in Emotionally and Ethically Sensitive Contexts

Paper Title

Breakdowns in Conversational AI: Interactional Failures in Emotionally and Ethically Sensitive Contexts

Publication Info

• Topic area: Interactional failures in conversational AI during emotionally and ethically sensitive dialogues.
• Keywords: Conversational AI, alignment, empathy, ethical guidance, multi-turn dialogue, interactional breakdowns, persona-conditioned simulation, emotional escalation, HCI, NLP.

Background and Problem

• Problem / challenge: Existing conversational AI systems struggle to maintain alignment in multi-turn, emotionally and ethically sensitive interactions. Prior work focuses on static or single-turn evaluations, neglecting dynamic interactional challenges.
• Significance: Failures in such contexts can harm user trust, emotional well-being, and ethical coherence, especially in scenarios involving moral ambiguity or escalating emotional tension.
• Motivation and related work: While reinforcement learning from human feedback (RLHF) and affective computing have improved safety and empathy, these approaches often operate in isolation. Safety-focused models may appear rigid, while empathy-focused models risk moral boundary violations. Current evaluation methods lack the ability to model dynamic, multi-turn interactions with escalating emotional and ethical tensions.

Solution

• Proposed approach: A persona-conditioned user simulator with staged emotional pacing to stress-test conversational agents in multi-turn, ethically and emotionally sensitive dialogues.
• Novelty:
  1. Development of a persona-conditioned simulator that generates psychologically coherent, multi-turn user behaviors.
  2. Introduction of an emotion pacing function to simulate escalating emotional trajectories.
  3. Taxonomy of interactional breakdowns, categorizing failures into affective misalignments, ethical guidance failures, and cross-dimensional trade-offs.
  4. Design implications for improving conversational AI alignment strategies.
• Procedure and key techniques:
  1. Persona construction: Extract structured psychological profiles from ethically salient dialogues using a six-dimension schema.
  2. Emotion pacing: Introduce a turn-level pacing function to simulate gradual emotional escalation.
  3. Evaluation: Use LLM-as-judge protocols and diversity metrics to assess tone, empathy, ethical guidance, and engagement across multiple chatbot models.
  4. Taxonomy development: Analyze failure cases to identify and categorize recurring breakdown patterns.

Results

• Concrete findings:
  • GPT-4o achieved the highest average scores across dimensions (7.8/10), while Cosmo-3B performed the worst (4.3/10).
  • Emotion pacing increased breakdown rates across all models, with higher-capability models like GPT-4o showing a 12.4% rise in failures under escalating emotional pressure.
  • Breakdown types included refusal looping, escalation mismatches, superficial empathy, inconsistent ethical guidance, and affective-ethical trade-offs.
• Advantage over baselines:
  • The simulator revealed systematic vulnerabilities in conversational agents that were not apparent in static or single-turn evaluations.
  • Emotion pacing exposed failures in ethical guidance and affective alignment, particularly in morally ambiguous or high-stakes scenarios.
• Experiments / evaluation:
  • Models tested: Cosmo-3B, Emotional-LLaMA-8B, Llama-2-7b-chat, Llama-3-8B-Instruct, GPT-4o.
  • Metrics: Respectful tone, ethical guidance, empathy, specificity and engagement, intra-dialogue diversity.
  • Scenarios: Six ethically salient categories (e.g., serious illegal acts, moral dilemmas, benign conversations).
• Limitations and future work:
  • Shared-model biases due to reliance on GPT-based components.
  • Limited granularity in emotion pacing for morally laden emotions like guilt or shame.
  • Scenario filtering may introduce noise; future work could involve domain-expert annotation or task-specific classifiers.

Summary

This study identifies systematic interactional failures in conversational AI during emotionally and ethically sensitive multi-turn dialogues. Using a persona-conditioned simulator with emotion pacing, the authors reveal recurrent breakdowns such as refusal looping, ethical guidance failures, and affective-ethical trade-offs. GPT-4o outperformed other models but still exhibited vulnerabilities under escalating emotional pressure. The findings emphasize the need for alignment strategies that integrate empathy and ethical clarity dynamically over the course of interactions. The proposed taxonomy and simulation framework provide actionable insights for improving conversational AI in high-stakes contexts.