Identifying Breakdowns in Conversational Recommender Systems using User Simulation
We present a methodology to systematically test conversational recommender systems with regards to conversational breakdowns. It involves examining conversations generated between the system and simulated users for a set of pre-defined breakdown types, extracting responsible conversational paths, and characterizing them in terms of the underlying dialogue intents. User simulation offers the advantages of simplicity, cost-effectiveness, and time efficiency for obtaining conversations where potential breakdowns can be identified. The proposed methodology can be used as diagnostic tool as well as a development tool to improve conversational recommendation systems. We apply our methodology in a case study with an existing conversational recommender system and user simulator, demonstrating that with just a few iterations, we can make the system more robust to conversational breakdowns.
Research Questions / Practical Problems
Question signals indexed for this paper.
Research Questions
3- How can user simulation techniques detect specific types of conversational breakdowns in conversational recommender systems?Category: Conversational and Dialogue-Based RecommendationSimilar questionsarrow_forward
- Can user simulation techniques help evaluate the robustness of conversational recommender systems and generate improvement suggestions?Category: Conversational and Dialogue-Based RecommendationSimilar questionsarrow_forward
- How can system failures, communication breakdowns, and dialogue flow interruptions in conversational recommender systems be classified and detected?Category: Conversational and Dialogue-Based RecommendationSimilar questionsarrow_forward
Practical Problems
1- Users often feel frustrated due to interruptions or misunderstandings when conversing with recommender systems.Category: Conversational and Dialogue-Based RecommendationSimilar questionsarrow_forward
- 80%
Let Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems
CHI '19· Explainable AI (XAI) +2
- 80%
Improving understandability of feature contributions in model-agnostic explainable AI tools
CHI '22· Explainable AI (XAI) +1
- 80%
Orbit: A Framework for Designing and Evaluating Multi-objective Rankers
IUI '25· Explainable AI (XAI) +1
- 67%
Evaluating the Utility of Conformal Prediction Sets for AI-Advised Image Labeling
CHI '24· Explainable AI (XAI) +2
- 67%
A decision-theoretic representation of assistive interfaces
CHI '26· AI-Assisted Decision-Making & Automation +2
- 67%
🌳-generAItor: Tree-in-the-loop Text Generation for Language Model Explainability and Adaptation
IUI '25· Explainable AI (XAI) +2
- 60%
UMLAUT: Debugging Deep Learning Programs using Program Structure and Model Behavior
CHI '21· Explainable AI (XAI) +1
- 60%
Contextualizing User Perceptions about Biases for Human-Centered Explainable Artificial Intelligence
CHI '23· Explainable AI (XAI) +1
- 60%
AutoML in The Wild: Obstacles, Workarounds, and Expectations
CHI '23· Explainable AI (XAI) +1
- 60%
Validating AI-Generated Code with Live Programming
CHI '24· Human-LLM Collaboration +1
Based on Jaccard similarity of research subtopics & professions (≥60%)