Scene-Aware Behavior Synthesis for Virtual Pets in Mixed Reality

Title of the Paper

Scene-Aware Behavior Synthesis for Virtual Pets in Mixed Reality

Paper Information

• Domain: Virtual pet behavior modeling and interaction in mixed reality environments
• Keywords: virtual pets, behavior synthesis, scene semantics, mixed reality, long short-term memory networks, user study
• Conference: CHI '21 (May 2021, Yokohama, Japan)

Research Background and Problem

• Background:

  • Virtual pets serve as substitutes for real pets, helping to alleviate loneliness and promote healthy lifestyles. However, traditional virtual pets lack environmental awareness and exhibit less natural behavior.
  • Mixed reality technology offers opportunities for enhanced immersion and natural interaction. Yet, enabling virtual pets to understand real-world scenes and perform reasonable behaviors within them remains a challenge.

• Research Questions:

  1. How can virtual pet behaviors resembling those of real pets be generated?
  2. How can virtual pets act reasonably within real-world environments?

• Significance:

  • Addressing these issues can enhance user immersion and broaden the applications of virtual pets in education, therapy, and entertainment.

• Related Work:

  • Previous studies primarily relied on hard-coded rules or randomly generated behaviors, which lack naturalness.
  • While behavior synthesis (robotics, gaming) and scene semantic understanding (e.g., object detection) have seen advancements, they have not been deeply integrated into virtual pet applications.

Proposed Solution

• Overview:

  • A scene-aware virtual pet behavior generation method is proposed, combining real-world scene semantic information to generate natural behavior sequences.
  • The approach includes two main modules: a data-driven behavior generator and a behavior instantiation module (for executing behaviors in physical environments).

• Innovations:

  • An LSTM model trained on real pet data is used to generate high-level behavior sequences, improving naturalness.
  • Scene semantic understanding (via Mask R-CNN) is introduced, enabling virtual pets to comprehend real-world objects and associate behaviors with object locations.

• Implementation Steps:

  1. Data-Driven Behavior Generator:
     • Collect real pet behavior data and annotate the relationships between behaviors and scene objects.
     • Train a two-layer LSTM network to capture behavior patterns and generate high-level behavior sequences.
  2. Scene Understanding:
     • Use mixed reality headsets (e.g., Hololens) to scan scenes and obtain 3D models and object information.
     • Detect objects in the scene and generate semantic information using Mask R-CNN.
  3. Behavior Instantiation:
     • Assign the generated behavior sequences to actual locations in the scene (e.g., sofa, table).
     • Use an improved A* algorithm to optimize the path from one behavior location to the next, ensuring the path aligns with real pet behavior patterns.

Research Outcomes

• Key Findings:

  • Successfully implemented a scene-aware virtual pet behavior generation and instantiation method.
  • The proposed method generates natural behaviors that adapt to real-world physical environments.

• Comparison with Existing Solutions:

  • Outperforms traditional rule-based or randomly generated behavior methods in terms of naturalness, diversity of behavior transitions, and adaptability to scenes.
  • Improved path planning better avoids obstacles and generates open paths preferred by pets.

• Experiments and User Study:

  • Validation in living room, bedroom, and kitchen scenarios:
    • Behavior generation outperformed probability sampling and random generation methods.
    • Behavior location assignment and path planning were significantly more reasonable compared to random methods and traditional path planning.
  • Users rated the naturalness of virtual pet behaviors and their interaction with the scene highly, often describing the application as "vivid" and "realistic."
  • In kitchen scenarios, due to shorter-term behaviors, differences with baseline methods were not statistically significant.

• Limitations:

  • Training based on datasets is limited to indoor scenarios, and outdoor behaviors cannot yet be generated.
  • Performance limitations of mixed reality devices led to some object detection failures (e.g., black objects or dynamic scenes).

• Future Directions:

  1. Incorporate more real-world datasets, including outdoor behavior data.
  2. Achieve low-level motion detail synthesis to enhance behavioral precision.
  3. Add user interactions such as voice and gestures.
  4. Extend the approach to robotic pets or virtual pets in video games.
  5. Address dynamic scenes to enable real-time path and behavior updates, improving system robustness.