UX Research on Conversational Human-AI Interaction: A Literature Review of the ACM Digital Library

Title of the Paper

UX Research on Conversational Human-AI Interaction: A Literature Review of the ACM Digital Library

Paper Information

• Field of Study: Human-Computer Interaction (HCI), User Experience (UX), Conversational AI
• Keywords: Conversational agents, chatbots, conversational AI, user experience research, literature review

Research Background and Questions

• Background: Conversational agents (CAs), such as chatbots, are rapidly evolving. Initially, they primarily supported dyadic (one-on-one) human-computer interaction. Recently, there has been a growing trend toward supporting polyadic interactions, enabling both human-computer and interpersonal interactions simultaneously. However, research on the design and evaluation of multi-user conversational agents, particularly their role in interpersonal interactions, remains scattered across various fields and lacks systematic synthesis.
• Research Questions:
  1. How do multi-user conversational agents address challenges in interpersonal interactions?
  2. What are the differences between research on dyadic and multi-user conversational agents?
  3. What are the best practices for designing multi-user conversational agents?
  4. What impact do these agents have on interpersonal interactions?
  5. What metrics are used to evaluate the user experience of multi-user conversational agents?
  6. What issues are overlooked in design research?
• Significance: Addressing the design and ethical challenges of multi-user conversational agents in interpersonal interactions is crucial for enhancing user experience and fostering innovation in applications.

Solutions

• Research Methods and Steps:

  • Conducted a systematic review of the ACM Digital Library, screening 1,302 articles and ultimately identifying 36 studies on multi-user conversational agents and 135 studies on dyadic interactions.
  • Employed a mixed-methods analysis combining qualitative and quantitative approaches, including topic modeling and open coding.

• Innovations:

  • Systematically summarized the key challenges that multi-user conversational agents can address in interpersonal interactions.
  • Proposed a comprehensive set of metrics for evaluating user experience.
  • Explored the social boundary issues in the design of multi-user conversational agents.

• Implementation Steps and Techniques:

  1. Data Collection: Filtered articles from the ACM database using inclusion/exclusion criteria, such as whether user experience evaluation was included and whether the study addressed multilingual contexts.
  2. Qualitative Analysis: Used thematic analysis to define the unique designs and practices of multi-user interactions.
  3. Quantitative Analysis: Applied topic modeling tools to uncover popular themes in research on multi-user and dyadic conversational agents.

Research Findings

• Challenges Addressed by Multi-User Conversational Agents:

  • Communication Efficiency: Mitigating issues such as disorganized communication structures and task management difficulties in collaboration.
  • Lack of Engagement: Encouraging group members to participate more equally and effectively.
  • Relationship Maintenance: Assisting teams in managing emotions, building trust, and preventing conflicts.
  • Building Connections: Facilitating cross-cultural "ice-breaking" conversations and initial relationship building.

• Differences in Research Focus:

  • Studies on dyadic agents focus more on the quality of individual interactions with AI.
  • Research on multi-user agents emphasizes improving interpersonal collaboration in scenarios such as team discussions, education, and online communities.

• Common Design and Evaluation Methods:

  • Design Methods: Most studies adopt framework-driven or user-participatory design approaches.
  • Common Metrics: Include task completion time, user engagement, perceived social support, and conversational fluency.
  • Evaluation Methods: Employ experiments, surveys, interviews, and log analysis. In multi-user scenarios, social behaviors and emotional dynamics are particularly considered.

• Unaddressed Issues in Systematic Research:

  • Multi-user conversational agents require clear "visibility" and "ignorable" design considerations.
  • "Boundary-awareness" should be embedded in agent design to address the complex relationships between privacy, ethics, and social dynamics.
  • Existing theories are rarely applied to the design of multi-user agents. Future research should integrate theories from sociology and psychology.

• Comparative Advantages Over Existing Solutions: This review is the first to systematically explore the role of multi-user conversational agents in user experience and social relationships, providing a robust theoretical foundation and practical recommendations for future design.

• Limitations and Future Directions:

  • The study is limited to the ACM database, with insufficient coverage of research from other sources.
  • Further exploration of innovative agent designs and real-world experimental evaluations is needed.
  • Future research should delve deeper into the ethical and social impacts of boundary-aware agents and address new dimensions of complexity in multi-user interaction scenarios.