DreamStream: Immersive and Interactive spectating in VR

Document Title

DreamStream: Immersive and Interactive Spectating in VR

Document Information

• Topic Area: Virtual Reality (VR), Human-Computer Interaction
• Keywords: Virtual Reality, Real-time Streaming, User Collaboration, Immersive Experience, Interactive Spectating, Depth Textures, Multiplayer, OpenVR, 3D Reconstruction, User Interface

Research Background and Problems

• Problems and Challenges:
  • Current VR activity spectating primarily relies on viewers watching 2D videos on traditional monitors, which limits the immersive experience for spectators.
  • Existing multi-user VR systems require spectators to download full VR applications, posing high device requirements and making "plug-and-play" experiences, such as live streaming, difficult to achieve.
  • There is no widely applicable solution to support streaming VR applications in 3D scene formats.
• Importance:
  • As VR becomes increasingly popular in gaming, collaboration, and media consumption, there is growing demand for more immersive spectating experiences.
  • Enhancing spectators' interactivity and experience can optimize the entertainment and educational value of VR content, as well as improve the social dynamics of live streaming or gaming.
• Research Motivation and Related Work:
  • Limitations of existing solutions:
    • 2D video streaming restricts the immersive experience for spectators.
    • Some multi-user VR systems require each spectator to install full applications, limiting convenient access to scenes.
  • An ideal solution should enable spectators to "observe" VR scenes immersively in 3D at low cost, while supporting real-world collaborative interactions.

Solution

• Main Methods and Framework:
  • Proposing the DreamStream system to provide spectators with real-time 3D reconstructed VR viewing experiences.
  • The system consists of three components:
    • DreamStream-Host: A module running on the VR player's device, responsible for intercepting and converting texture data (RGB and depth) from VR applications, generating 2.5D video frames embedded with depth information.
    • DreamStream-Relay: A relay service running on cloud or local servers, responsible for coordinating multi-user data stream distribution.
    • DreamStream-Client: A client running on spectator devices, used to reconstruct VR scenes and render interactive 3D interfaces.
• Technology and Implementation Steps:
  1. Intercepting Video and Depth Textures: Using OpenVR library function calls to capture RGB and depth buffers from VR applications.
  2. Generating 2.5D Video Frames: Encoding texture and depth data into H.264 video streams.
  3. Secondary Viewpoint Sampling: Simulating third-person perspectives by altering the VR user's headset position to capture surrounding environmental information in the VR scene.
  4. Streaming and Rendering:
     • The Relay acts as a central management node, coordinating interactions between multiple clients and the host.
     • Spectator clients use 2.5D streams to reconstruct and render VR scenes.
  5. Supporting Interaction and Synchronization:
     • Adding virtualized real-time 3D avatars for spectators and players to enhance collaboration.
     • Implementing tools like laser pointers to facilitate interaction between spectators and players.
• Innovations:
  • Achieves real-time 3D streaming across applications without modifying existing VR application code.
  • Decouples the perspectives of players and spectators through the combination of depth buffers and RGB textures.
  • Supports VR/2D dual-mode viewing and real-time multi-user collaboration.

Research Outcomes

• Specific Results:
  • Testing validated the applicability of the DreamStream system in seven popular VR games (e.g., Skyrim, TiltBrush, BeatSaber).
  • User evaluations showed that DreamStream provides a more immersive and interactive spectating experience.
• Experimental Results:
  • Compared to baseline systems, participants rated DreamStream higher in terms of scene comprehension, immersion, and collaboration.
  • Statistically, using scales such as NASA-TLX to measure cognitive load, the DreamStream system significantly reduced workload.
• Advantages:
  • Provides a streaming-based "plug-and-play" spectating experience.
  • Standardized encoding of audio-video streams improves performance, enabling high-quality, low-latency viewing for multiple spectators.
  • Enhances interaction between spectators and VR players in traditional asymmetric collaborative environments.
• Limitations:
  • 3D scene reconstruction may exhibit artifacts, especially during rapid movements and at complex image edges.
  • Due to the computational intensity of real-time streaming, encoding frame rates may occasionally drop (i.e., FPS reduction).
  • Global synchronization and scene management for multi-spectator support still require improvement.
• Future Directions:
  • Improving 3D reconstruction algorithms to reduce artifacts.
  • Investigating how to support large-scale spectator interactions in social dynamics.
  • Evaluating the adaptability of different types of VR applications (non-gaming scenarios such as education and collaboration).
  • Exploring automated methods for optimizing streaming quality (e.g., real-time adjustments based on caching).

Reference Analysis

• This work not only proposes an easy-to-use VR spectating system but also advances technological improvements in the fields of VR collaboration and real-time interaction.
• Through specific designs (e.g., real-time depth texture extraction, Spectator Avatar system), it effectively addresses the user experience deficiencies of existing VR streaming systems.