Conversational Agents on Your Behalf: Opportunities and Challenges of Shared Autonomy in Voice Communication for Multitasking

Research Background and Issues

• What problems or challenges did the authors identify?
  The authors explored how conversational agents can support users in multitasking during real-time voice communication. While fully autonomous conversational agents can reduce user burden and enhance their ability to complete other tasks, relinquishing full control to the agent may lead to discomfort or deviations from user expectations. The shared autonomy model aims to balance task delegation and control, but its impact on multitasking remains unclear.

• Why is this issue important?
  Multitasking is a common phenomenon in modern work and life, but it often reduces cognitive efficiency. Conversational agents can assist users in staying engaged while completing additional tasks, thereby improving productivity. This has significant implications for corporate environments, educational settings, and more.

• Research Motivation and Related Work
  Previous studies have shown that hybrid autonomy and human-machine collaboration (e.g., shared autonomy in robotic operations) can enhance efficiency. However, their application in real-time voice communication and multitasking scenarios has not been systematically validated. The authors further investigated how shared autonomy agents affect user performance and experience.

Solution

• What methods or solutions did the authors propose?
  The authors designed an experimental multitasking communication system that allows a conversational agent to assist users in multitasking through three levels of autonomy (no autonomy, full autonomy, shared autonomy). The agent interacts with users and remote partners by predicting language activity, performing real-time speech transcription, generating responses, and managing context.

• What is innovative about this solution?
  Unlike traditional fully autonomous systems, the shared autonomy model allows users to participate in conversations as needed. This design seeks to balance task delegation and control while leveraging advanced language processing technologies (e.g., Voice Activity Projection models and the GPT-4 language model) to generate user-aligned responses.

• What are the implementation steps and key technologies used?

  1. Develop a communication system with three levels of autonomy: no autonomy, full autonomy, and shared autonomy.
  2. Manage agent conversation responses through pre-recorded audio and real-time language generation to ensure contextual relevance.
  3. Design an experimental process where participants simultaneously complete a communication task (evaluating student profiles) and a complex arithmetic task.
  4. Use tools such as Zoom, Python scripts, and AI models to enable voice interaction and real-time processing.

Research Findings

• What specific findings were obtained?

  • The full autonomy mode significantly improved users' performance on secondary tasks (arithmetic tasks).
  • Remote partners did not perceive differences in agent involvement, and conversation quality ratings were similar across modes.
  • While the shared autonomy mode increased user control over conversations, it did not significantly enhance multitasking performance.

• What advantages does it offer compared to existing solutions?

  • Provides a flexible shared autonomy mechanism, allowing users to intervene in system conversations as needed rather than relying entirely on the agent.
  • Achieves a comprehensive evaluation of conversation quality and multitasking performance, offering new design insights for the field.

• What were the experimental or evaluation results?

  • Data showed that the full autonomy mode significantly reduced users' cognitive load and improved secondary task performance (33% more questions answered with stable accuracy).
  • The shared autonomy mode increased users' subjective satisfaction with conversations, but their arithmetic task performance declined due to the cognitive effort required to monitor the agent's dialogue.

• Limitations and Future Directions

  • Limitations:
    • The experimental setup was relatively simple and did not test the system's effectiveness in more complex or high-stakes scenarios, such as unstructured meetings in higher education or corporate settings.
    • Users' initial unfamiliarity with the system may have caused discomfort or monitoring behaviors, which could improve over time with increased familiarity.
    • The current agent's response quality remains suboptimal, occasionally leading to contextual mismatches.
  • Future Directions:
    • Explore the design of shared autonomy systems in more complex task scenarios, such as education, large-scale remote meetings, and customer service.
    • Investigate diverse system applications beyond the observed tasks, such as visual feedback support or dynamic behavior enhancement based on historical data.
    • Expand the sample population for long-term usage tracking to analyze potential changes in usage habits and societal acceptance of such technologies.

Conclusion

This study provides valuable empirical evidence and design guidelines for developing shared autonomy conversational agents, particularly in addressing the challenges of balancing user autonomy and agent functionality in multitasking contexts. However, achieving smarter and more flexible conversational agents will require further technological advancements and a deeper understanding of user needs.