How to Guide Task-oriented Chatbot Users, and When: A Mixed-methods Study of Combinations of Chatbot Guidance Types and Timings

Document Title

How to Guide Task-oriented Chatbot Users, and When: A Mixed-methods Study of Combinations of Chatbot Guidance Types and Timings

Document Information

• Subject Area: Optimization of User Interaction with Task-oriented Chatbots
• Keywords: Chatbots, Laboratory Study, Task Completion, User Guidance, Interaction Breakdown, Tool-based Intelligent Systems, Mixed Methods

Research Background and Issues

• Problems or Challenges:

  • Ensuring smooth conversations between users and chatbots is a significant challenge, often disrupted by the chatbot's misunderstanding of user intent or inability to recognize certain expressions.
  • Users tend to overestimate or misunderstand the chatbot's capabilities, leading to operational issues.
  • Existing research suggests providing specific guidance for system interaction, but there is debate over when and how to offer such guidance.
  • There is a lack of empirical studies on the types of guidance content and their timing.

• Importance:

  • Chatbots are increasingly popular, and task-oriented bots can save users time on repetitive tasks, improving efficiency.
  • Understanding how the content and timing of guidance affect user experience and task completion is fundamental to improving chatbot design.

• Research Motivation:

  • There is limited research on the combined effects of guidance types (example-based and rule-based) and delivery timings (initial service, pre-task, post-failure, on-demand).
  • Clarifying the impact of these combinations on task completion efficiency, learning improvements across consecutive tasks, and subjective user experience.

Research Objectives

This study aims to address the following key questions:

1. Can combinations of different guidance types and timings improve users' task completion efficiency, conversation progress, and subsequent task performance?
2. How do users subjectively evaluate these combinations?
3. What are the characteristics of the guidance types and timings that users prefer?

Solutions

• Research Methods:

  • Proposed eight combinations of two guidance types (example-based and rule-based) and four guidance timings.
  • Employed a mixed-methods approach with two main phases:
    • Quantitative Experiment: Examined the impact of guidance types and timings on user task performance.
    • Qualitative Reflection Study: Conducted interviews to understand user preferences and subjective perceptions of different guidance strategies.

• Key Techniques and Steps:

  • Developed a chatbot based on IBM Watson, supporting travel and movie booking scenarios with varying task complexities.
  • Controlled variables in the experiment and analyzed task success rates, completion times, and learning effects using mixed regression models.
  • Used affinity diagramming for thematic synthesis of qualitative data.

• Innovations:

  • Conducted innovative quantitative and qualitative analyses of common guidance method combinations.
  • Systematically analyzed the combined effects of guidance content and timing on users' short-term efficiency and long-term learning.
  • Provided practical recommendations for designing guidance in task-oriented chatbots.

Research Outcomes

Key Findings

1. Task Performance (Quantitative):

   • Example-based guidance performed better in initial tasks but showed weaker learning improvements across tasks.
   • Rule-based guidance, though slower initially, resulted in significant learning effects and improved subsequent task performance.
   • Providing examples during task introduction significantly reduced task completion time, while offering rules post-failure helped prevent interaction breakdowns.

2. User Subjective Preferences:

   • Example-based guidance during task introduction was highly rated by users, who found it natural and intuitive.
   • Rule-based prompts post-failure received the lowest ratings, as users sometimes perceived the rules as "incorrect information," negatively impacting the experience.

3. Learning Improvements and Long-term Effects:

   • Rule-based guidance helped users build accurate mental models of the system, facilitating learning.
   • Offering example-based guidance on-demand encouraged users to actively explore and learn.

Design Recommendations

• Hybrid Use:

  • Provide example-based guidance during task introduction and supplement with rule-based prompts post-failure.
  • For complex tasks or first-time users, adopt gradual guidance tailored to the task context.

• Avoid "Ineffective Displays":

  • Initial guidance content should be concise and effective to avoid user disengagement with lengthy text.
  • Post-failure guidance should avoid overly formal tones to reduce user frustration.

• Future Directions:

  • Enhance guidance features aimed at user learning, particularly focusing on the transferability of guidance.
  • Explore more adaptive recommendation algorithms tailored to flexible user contexts.

Limitations and Future Directions

• Limitations:

  • The study was conducted in a controlled laboratory environment, and the data may not fully reflect real-world usage.
  • The sample was concentrated on young, urban users, and generalizability needs further validation.
  • Task complexity was limited to two scenarios; future research could extend to broader application domains (e.g., business and technical support).

• Future Research:

  • Expand task types and participant demographics to explore the generalizability and applicability of guidance across domains.
  • Investigate the additional impact of detailed guidance settings (e.g., guidance language style, visual presentation) on users.