Cooking With Agents: Designing Context-aware Voice Interaction

Document Title

Cooking With Agents: Designing Context-aware Voice Interaction for Complex Tasks

Document Information

• Subject Area: Human-Computer Interaction, specifically the design and implementation of voice assistants in complex task interactions.
• Keywords: Voice interface, conversational user interface, cooking, conversational analysis, task complexity, context awareness, intelligent voice assistant, user experience, artificial intelligence, human-machine dialogue

Research Background and Issues

• Issues and Challenges:

  1. Current voice assistants (VAs) have limited conversational capabilities, particularly in adapting to multi-turn dialogues and user social behaviors.
  2. VAs lack sufficient context awareness, leading to issues such as irrelevant information, misunderstood requests, or information overload—especially in complex tasks requiring temporal and spatial coordination (e.g., cooking).
  3. Designing context-aware VAs to better support users in completing complex and time-sensitive tasks remains underexplored.

• Significance:

  • Cooking, as a typical complex task, imposes high demands on user interaction with intelligent assistants. Studying VAs in such scenarios can expand their application scope and improve user experience.

• Research Motivation and Related Work:

  1. Current research focuses on simple tasks (e.g., playing music, retrieving information) or specific conversational patterns, with limited exploration into context-aware design for VAs in complex tasks.
  2. In cooking scenarios, studies show users rely more on audio output, but the limitations of current VAs significantly impact the fluency of continuous dialogue and task support.
  3. The recent development of generative AI models and large language models (LLMs) provides technical support for context awareness and memory capabilities, yet practical implementation and interaction design require further experimentation and validation.

Solution

• Methods and Solutions:

  1. Preliminary Study: Conduct user testing with Google Assistant (GA) to analyze the limitations of current commercial intelligent voice assistants in cooking scenarios (e.g., issues with context alignment).
  2. Improved Experiment: Design a context-aware VA system and simulate system responses using the Wizard-of-Oz method to observe changes in user interaction.
  3. Design Improvements:
     • Shared task vocabulary: Semantic matching of objects and actions in the task based on recipe text.
     • Task state tracking: Simulating VA's ability to understand the current task state and provide targeted feedback.
     • Proactive suggestions: Exploring how timely, task-related suggestions from the VA influence user experience.

• Innovations:

  • Combining context-aware functionality with the conversational capabilities of voice assistants to explore user behavior patterns and interaction simplification in multi-turn dialogues.
  • Developing a shared-context virtual interaction model and generating observable experimental data.

• Implementation Steps and Key Technologies:

  1. In the first phase (commercial voice assistant testing), record actual user interactions with Google Assistant.
  2. In the second phase, construct a context-aware "virtual assistant" experimental environment and simulate context-aware dialogue through manual control (Wizard of Oz).
  3. Analyze user language complexity, query efficiency, and social interaction behaviors with the assistant during interactions.

Research Results

• Specific Findings:

  1. Improved Context-aware Dialogue: Context-aware VAs significantly reduce user effort and dialogue complexity when querying or confirming information.
  2. User Behavior Patterns: Conversations with context-aware VAs are more complex and natural, including more queries based on shared context and explicit confirmations of task status.
  3. Proactivity Experiments: For time-sensitive tasks (e.g., setting timers), context-aware proactive features balance task completion efficiency and user autonomy.

• Comparison with Existing Solutions:

  • Compared to commercial voice assistants like Google Assistant, context-aware VAs significantly improve relevance and responsiveness in supporting complex tasks.
  • Users prefer engaging in semantically rich, multi-step interactions in context-aware systems rather than relying on single keyword commands.

• Experimental or Evaluation Results:

  • Across 16 Wizard-of-Oz tests, the average interaction completion time with context-aware VAs was 30 minutes, with overall task completion efficiency improved.
  • Video and corpus analysis revealed high user communication proactivity and a significant reduction in corrective dialogues (e.g., repeated instructions).

• Limitations and Future Directions:

  1. Limitations:
     • The simulated "context-aware" capabilities in the experiment were manually operated and not fully automated.
     • The proactive design of the system may cause distraction or interference in certain situations.
  2. Future Research:
     • Explore how multimodal large language models (e.g., GPT-4/Vision) can be leveraged to enhance voice assistants' context understanding capabilities.
     • Design interaction models that better balance user personal intentions (e.g., playing music) with specific task requirements.
     • Investigate how users manage and switch context constraints with voice assistants in multi-task scenarios.

Conclusion

This study demonstrates the potential improvements of context-aware VAs in complex tasks (e.g., cooking scenarios) compared to commercial voice assistants. It provides both theoretical and practical insights into human-machine dialogue design, emphasizing the importance of task context in optimizing user experience.