Collaboration with Conversational AI Assistants for UX Evaluation: Questions and How to Ask them (Voice vs. Text)

Title of the Paper

Collaborating with Conversational AI Assistants for UX Evaluation: Questions and Their Modalities (Comparison of Voice and Text)

Paper Information

• Field of Study: Human-Computer Interaction (HCI) and User Experience (UX) Analysis
• Keywords: User Experience (UX), UX Evaluation, Usability Testing, Human-AI Collaboration, Conversational Assistants, Text Assistants, Voice Assistants

Research Background and Issues

• Identified Problems or Challenges:

  • User experience (UX) analysis often requires reviewing usability testing audio and video recordings, a process that is time-consuming, labor-intensive, and challenging.
  • Current mainstream AI-assisted analysis tools primarily present evaluation information in non-interactive visual formats, which fail to meet the dynamic questioning needs of UX evaluators.
  • Collaborative practices for UX evaluation are limited, mainly due to the high cost and limited effectiveness of traditional team-based analysis.
  • Although machine learning-based models have been used to assist UX evaluation, these methods are still limited in capturing specific issues.

• Significance of the Research:

  • Conversational AI assistants enable natural language interaction to answer questions, which could improve analysis efficiency and provide UX evaluators with greater autonomy.
  • The demand for automation in UX analysis is increasing, but it still requires a combination of human critical thinking and AI efficiency to overcome the limitations of automated methods.

• Research Motivation and Related Work:

  • Current research focuses primarily on non-interactive AI-based analysis tools, with limited studies on the application of conversational interaction in UX analysis.
  • People tend to prefer natural language dialogue, perceiving AI as "another colleague," but existing tools have not fully explored the needs and potential of interactive analysis assistants.
  • Comparing the behavioral differences between voice and text interaction in the context of UX analysis can help design interaction tools that better meet user needs.

Proposed Solution

• Proposed Solution:

  • This study designs and implements a UX analysis tool based on a conversational AI assistant. Through a Wizard-of-Oz study in two interaction modes (voice and text), it explores the types of questions UX evaluators ask and their interaction preferences during UX analysis.
  • It compares the differences in user behavior and evaluations between voice and text assistants, extracting user design requirements for conversational assistants in UX analysis.

• Innovations:

  • Proposes an interaction model based on conversational AI assistants to meet the diverse questioning needs in UX analysis.
  • Systematically analyzes five major categories of questions that UX evaluators may ask and provides corresponding design suggestions for the first time.
  • Conducts an in-depth comparison of interaction modes between voice and text assistants, addressing a gap in existing research.

• Implementation Steps and Key Techniques:

  1. Designed an interface to simulate an AI assistant, including a video player and text/voice interaction windows.
  2. Used the Wizard-of-Oz method to simulate real-time responses to UX evaluators' questions. Assistant responses were based on a pre-built knowledge base and manual answers by researchers.
  3. Conducted experiments with 20 participants (10 for each interaction mode), recording and analyzing their interactions with the assistant and subjective evaluations of the assistant.
  4. Classified and coded 325 collected questions, analyzing differences in question types across the two interaction modes.

Research Findings

• Specific Findings:

  • Questions asked by UX evaluators can be categorized into five major types:
    1. User Actions: For example, the number of clicks or time spent on a page.
    2. User Mental Models: Including user perceptions, emotions, and reasons behind behaviors.
    3. Requests for AI Assistance: Such as asking for suggestions, querying functions, or controlling volume.
    4. Product and Task Information: Including product background information and ideal task paths.
    5. User Demographics: Including user backgrounds and participant information.
  • Text assistants outperformed voice assistants in the number of questions asked, but voice assistants were more interactive and engaging.
  • Voice assistants' audio responses could interfere with video playback, while text assistants demonstrated better efficiency.

• Advantages:

  • Achieved human-AI collaboration: AI provided data and suggestions, while UX evaluators made final judgments based on context.
  • Text and voice assistants each demonstrated unique strengths in efficiency and interactive experience, offering valuable insights for future tool design.

• Experimental or Evaluation Results:

  • Text assistants were rated as more efficient, particularly in scenarios requiring quick and accurate answers.
  • While voice assistants required more cognitive effort, some participants found voice interaction to be more natural and collaborative.
  • Participants expressed high satisfaction and trust in the assistants' responses, especially for objective questions.

• Limitations and Future Directions:

  • The number of videos used for data collection was limited (only two videos). Future research should expand the scope to include more interaction types and tasks.
  • The current experiment focused on short-term usage scenarios; further research is needed to explore question types and interaction behavior changes over long-term use.
  • Investigate designs that integrate automated visualization with conversational interaction to balance objective data summarization and subjective question answering.