Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agents

Paper Title

Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agents

Publication Info

• Topic area: Improving user feedback quality in conversational agents through design and interaction scaffolds.
• Keywords: Conversational agents, user feedback, feedback barriers, human-AI interaction, scaffolding, feedback quality, design desiderata, multi-turn tasks, large language models, interaction design.

Background and Problem

• Problem / challenge: Human feedback to conversational agents (CAs) is often sparse, vague, and low-quality, limiting the agents' ability to align with user goals and improve their performance. Existing systems lack mechanisms to elicit structured, actionable, and high-quality feedback.
• Significance: High-quality feedback is critical for improving conversational agents' alignment with user goals, especially in multi-turn, goal-oriented tasks. Addressing feedback barriers can enhance the effectiveness of human-AI collaboration.
• Motivation and related work: Prior research has explored feedback in human-AI interaction, emphasizing mixed-initiative systems, interactive machine learning, and explanatory debugging. However, these approaches often fail to address the challenges of multi-turn, open-ended tasks with LLM-powered agents. This paper builds on these efforts by identifying feedback barriers and designing scaffolds to overcome them.

Solution

• Proposed approach: FeedbackGPT, a system incorporating model-agnostic scaffolds designed to address feedback barriers and improve the quality and frequency of user feedback in conversational agents.
• Novelty:
  1. Identification of four key Feedback Barriers: Common Ground, Verifiability, Communication, and Informativeness.
  2. Development of design desiderata to mitigate these barriers: fostering a shared frame of reference, enabling proactive interaction, and providing transparent reasoning.
  3. Implementation of FeedbackGPT with seven scaffolds to operationalize the design desiderata and improve feedback quality.
• Procedure and key techniques:
  • Conducted a formative study with 16 participants to identify feedback barriers using Grice’s maxims.
  • Designed FeedbackGPT with scaffolds like inline comments, undo/redo, feedback huddle, quick actions, feedback evaluation, explanation, and split-view comparison.
  • Evaluated FeedbackGPT in a within-subject study with 20 participants, comparing it to ChatGPT on co-writing tasks.

Results

• Concrete findings:
  • FeedbackGPT increased goal-referenced feedback by 25.94% and actionable feedback by 24.76%.
  • Feedback volume (characters per turn) increased significantly from 242.33 to 585.06.
  • No significant improvement in feedback clarity, with vague inputs persisting.
• Advantage over baselines: FeedbackGPT enabled more goal-referenced, actionable, and progressive feedback compared to ChatGPT, though it introduced higher cognitive load for users.
• Experiments / evaluation:
  • Study 1: Formative interviews with 16 participants to identify feedback barriers.
  • Study 2: Within-subject comparison of FeedbackGPT and ChatGPT with 20 participants performing co-writing tasks. Metrics included feedback quality (goal-referenced, actionable, articulated, progressive) and subjective user experience.
• Limitations and future work:
  • Limited participant diversity and potential selection bias.
  • Focus on co-writing tasks; results may not generalize to other domains.
  • Short-term study; user behavior over time remains unexplored.
  • Scaffolds represent one implementation of the design desiderata; further exploration of alternative designs is needed.

Summary

This paper identifies four key feedback barriers in conversational agents—Common Ground, Verifiability, Communication, and Informativeness—and proposes FeedbackGPT, a system with scaffolds designed to mitigate these barriers. FeedbackGPT significantly improved goal-referenced, actionable, and progressive feedback compared to a baseline (ChatGPT), though it increased cognitive load for users. The findings highlight the importance of interaction design in eliciting high-quality feedback and call for advancements in LLM capabilities to further support effective human-AI collaboration. Future work should explore longitudinal studies, diverse tasks, and alternative scaffold designs.