Neighbor-Environment Observer: An Intelligent Agent for Immersive Working Companionship

Document Title

Neighbor-Environment Observer: An Intelligent Agent for Immersive Working Companionship

Document Information

• Subject Area: Applications of human-computer interaction and artificial intelligence in virtual and physical environments
• Keywords: Virtual reality, interactive systems, collaborative agents, smart home, mixed reality, human-machine symbiosis, user state recognition

Research Background and Issues

• Research Questions and Challenges:

  1. When users are immersed in a virtual working environment, physical disruptions (e.g., phone ringing or visitors) may interrupt the sense of immersion;
  2. Most current immersive systems lack comprehensive control over both physical and virtual environments, failing to effectively integrate the two;
  3. How to enable artificial agents to observe, understand, and assist users in tasks across physical and virtual environments remains an unresolved issue.

• Significance:

  1. As virtual reality technology becomes increasingly widespread, optimizing user experience during transitions between virtual and physical environments is crucial for productivity enhancement;
  2. The absence of appropriate cross-environment coordination mechanisms reduces user acceptance of immersive experiences.

• Research Motivation and Related Work: The authors analyzed existing work, including AI applications in virtual environments, user state understanding, and typical feedback mechanisms in smart homes, but identified a lack of complete implementation of the concept of "Symmetrical Reality."

Solution

• Proposed System and Method: The authors introduced an intelligent agent system named "Neighbor-Environment Observer" (NEO). NEO utilizes non-invasive sensors to observe physical and virtual worlds in real-time, providing seamless assistance to immersive users.

• Key Innovations:

  1. Joint Observation: Integration of physical and virtual environment information to capture environmental changes in real-time;
  2. Scalable Decision-Making Algorithm: Supports automated decision-making, user preference learning, and diverse sensor configurations;
  3. Joint Action: Real-time execution of actions combining physical and virtual avatars;
  4. Personalization: Adjusts behavior based on user feedback to align with individual preferences.

• Implementation Steps and Key Technologies:

  1. Perception Module: Utilizes sensors such as RGB cameras and microphones to collect data on the physical environment, user state, and virtual environment;
  2. Decision Module: Employs And-Or graphs and probabilistic models to infer user states and generate action decisions;
  3. Action Module: Completes complex tasks, such as delivering items or notifying users of battery status, through virtual graphical interactions and physical robotic entities;
  4. The system prototype is implemented using wireless data transmission and standard devices (e.g., HTC Vive), with high scalability.

Research Outcomes

• Specific Outcomes:

  1. Developed a prototype of the NEO system operating in a standard smart home environment;
  2. Achieved functionalities of joint observation and autonomous decision-making, validating NEO's ability to provide real-time support in complex dynamic scenarios;
  3. Enhanced user experience through personalized feedback.

• Comparison with Existing Solutions:

  1. Compared to existing VR and smart home agent systems, NEO significantly improves coordination between physical and virtual environments;
  2. The system introduces the concept of symmetrical reality, surpassing single-environment intelligent agents at the methodological level.

• Experimental or Evaluation Results:

  1. System Evaluation 1: Assessed the impact of different sensor groups (physical environment, virtual environment, etc.) on system decision accuracy, confirming the necessity of joint observation;
  2. System Evaluation 2: Tested the system's ability to learn user preferences, demonstrating rapid adaptation and learning in three distinct user scenarios;
  3. User Study:
     • User focus significantly improved compared to scenarios without assistance;
     • Workload notably reduced, especially in situations involving physical disruptions;
     • User experience scores (UEQ-S) were high, with further optimization after personalization.

• Limitations and Future Directions:

  1. Due to current implementation, NEO cannot handle certain complex disruptions (e.g., spilled water);
  2. Personalization delays may lead to unexpected decisions;
  3. Future work will focus on supporting multi-user scenarios, expanding hardware (e.g., robotic arms and additional sensors), and exploring special cases in multi-virtual environments.