An Evaluation of Situational Autonomy for Human-AI Collaboration in a Shared Workspace Setting

Title of the Paper

Evaluating the Impact of Contextual Autonomy in Shared Workspaces: Human-AI Collaboration

Paper Information

• Subject Area: Human-Computer Interaction (HCI), AI Collaboration in Shared Workspaces
• Keywords: Human-AI Interaction, Shared Workspaces, AI Autonomy, Human-AI Collaboration, Task Allocation, Adaptive Autonomy

Research Background and Problem Statement

• Problems and Challenges:

  • Collaboration in Human-AI Teams (HATs) for highly complex tasks remains significantly challenging.
  • Excessive AI autonomy does not necessarily improve team performance; adjusting AI autonomy according to context might be more effective, but there is a lack of systematic empirical studies to support this.
  • Most related research focuses on multi-agent systems rather than shared workspaces involving real human users.

• Significance:

  • Human-AI collaboration holds significant application potential in fields such as industrial automation, collaborative office assistance, and intelligent driving.
  • Adjusting AI autonomy could have a critical impact on task execution efficiency and user experience.

• Motivation and Related Work:

  • Existing studies suggest that dynamically adjusting AI autonomy (e.g., based on user feedback) can help with specific tasks but have not deeply explored the empirical impact of dynamic autonomy on team collaboration tasks in shared workspaces.
  • Based on theoretical derivations and related work, there is an urgent need to develop a more rigorous experimental framework to study the pros and cons of adjusting AI autonomy.

Proposed Solution

• Proposed Solution:

  1. Design an experimental environment simulating a shared workspace to evaluate AI performance under fixed autonomy and contextually adaptive autonomy.
  2. Propose a set of general autonomy adjustment standards and derive adjustment rules suitable for the experimental environment based on theory and small-scale experiments.
  3. Measure task performance and user satisfaction under different AI autonomy modes.

• Innovations:

  • For the first time, systematically study the impact of AI contextual autonomy on team performance and user experience in real user tasks.
  • Incorporate the "Theory of Mind Models" (ToMMs) technique to enable AI to predict human behavior and adjust its autonomy accordingly.
  • Provide a unified experimental design framework to facilitate reproducibility and expansion of similar future research.

• Implementation Steps and Key Technologies:

  1. Construct a collaborative scenario based on "office joint tasks," requiring AI and humans to work together to organize documents and classify books.
  2. Define four autonomy modes: no autonomy (following instructions only), low autonomy (moderate questioning), medium autonomy (making suggestions and waiting for confirmation), and high autonomy (acting independently and notifying afterward).
  3. Introduce a "contextual adaptive autonomy" mode that dynamically switches autonomy levels based on real-time task demands (achieved through five key criteria, such as AI confidence and human behavior prediction).

Research Findings

• Specific Findings:

  • Teams using the adaptive autonomy AI mode achieved an average score of 220, outperforming all fixed-mode conditions.
  • Under the dynamic adjustment condition, users rated AI intelligence significantly higher and reported the highest satisfaction with collaboration and team member perception.
  • While the high-autonomy mode performed well, users exhibited low tolerance for its errors (e.g., providing incorrect labels).

• Advantages Over Existing Solutions:

  • Dynamic autonomy adjustment mitigated the negative user experience caused by errors in high-autonomy AI.
  • Automatic AI adjustments reduced the number of user intervention operations, allowing users to focus more on their own tasks.

• Experimental or Evaluation Results:

  1. Task completion efficiency was highest under the dynamic adjustment condition, with file and book classification numbers exceeding those in fixed modes.
  2. Only under the dynamic adjustment mode was the AI perceived as "intelligent" rather than merely a tool for execution.
  3. Users requested more autonomous AI behavior only under the low fixed-autonomy condition, whereas in high-autonomy and dynamic adjustment modes, users preferred lower adjustment costs (e.g., fewer incorrect labels).

• Limitations and Future Directions:

  • Limitations:
    • Interaction control design (e.g., switching between keyboard and mouse) increased user costs and sometimes caused missed AI prompts.
    • Although the experimental tasks simulated real collaborative scenarios, they might not fully reflect more complex real-world tasks.
    • The AI prediction model (e.g., user behavior prediction) was based on limited rules and did not yet integrate learning algorithms.
  • Future Directions:
    1. Introduce more complex task settings to study how high-risk errors affect team performance.
    2. Explore more advanced dynamic adjustment methods, such as real-time learning and prediction-based advanced autonomy models.
    3. Develop multimodal interaction methods (e.g., voice feedback) to reduce user information burden and enhance task efficiency.

Conclusion

By studying the impact of dynamic contextual autonomy AI on team performance and human experience in shared workspaces, this paper validates the potential of dynamic autonomy in HATs, particularly its significant advantages in improving team efficiency and perceived AI intelligence. This research lays a new foundation and direction for the future design of human-AI collaboration systems.