Tele-Immersive Improv: Effects of Immersive Visualisations on Rehearsing and Performing Theatre Online

Document Title

Tele-Immersive Improv: Effects of Immersive Visualisations on Rehearsing and Performing Theatre Online

Document Information

• Subject Area: Remote collaboration technologies and theatrical performance applications
• Keywords: tele-immersion, tele-presence, improvisation, immersive communication, mixed reality, creativity, digital theatre, flow states, virtual environments, video conferencing

Research Background and Issues

• Issues and Challenges:

  • Social restrictions during the COVID-19 pandemic prevented theatre actors from conducting face-to-face rehearsals and performances.
  • Popular video conferencing tools (e.g., Zoom, Skype) are primarily designed for business and education purposes and fail to meet the specific needs of theatrical training and performance.
  • Although remote collaboration tools have been studied in the past, there is limited research directly addressing how video conferencing impacts actors' improvisational experiences in isolated environments.

• Significance:

  • Improvisational theatre relies on participants' real-time responsiveness and collaborative storytelling.
  • The limitations of current remote technologies (e.g., lack of body language and co-presence) significantly affect performance quality, making research on addressing these challenges highly valuable.

• Research Motivation and Related Work:

  • Tele-immersive systems have been shown to enhance users' sense of presence, but most studies focus on controlled laboratory environments, with little exploration of practical applications for home users.
  • Existing research has left a gap in exploring "the impact of immersive communication on theatre," and this study aims to fill that void.

Solution

• Methods and Solutions:

  • Proposed and developed a media server called "Virtual Director."
  • This tool combines video conferencing with immersive 3D virtual scenes, providing actors with a dynamic virtual environment for real-time interaction.
  • Through background segmentation and scene composition, users can see themselves and remote partners sharing the same virtual space, integrating 2D video with 3D environments.

• Innovations:

  1. Enables immersive experiences for remote users without additional hardware.
  2. Offers dynamic backgrounds and real-time user interaction, significantly enhancing the sense of presence in remote performances.
  3. Specifically designed for theatrical rehearsals and performances, reducing visual distractions common in traditional video conferencing.

• Implementation Steps and Key Technologies:

  • Video input and background segmentation: Extracts each participant's video stream and uses real-time chroma keying to remove backgrounds.
  • 3D modeling and dynamic manipulation of virtual scenes: Embeds video streams into virtual scenes, allowing real-time adjustments to position, scale, and rotation.
  • User interface and operation: Provides an intuitive control interface, enabling flexible background and object adjustments via MIDI controllers.
  • Developed core functionalities using the TouchDesigner platform, supporting NDI video standards for seamless video stream transmission.

Research Outcomes

• Specific Results:

  • Actors widely reported stronger feelings of connection and physical presence with their peers in the virtual shared space.
  • Experiments demonstrated that actors found it easier to enter the "flow" state required for improvisational theatre when seeing themselves and remote partners placed in the same virtual scene.
  • Dynamic backgrounds not only replaced traditional "scene descriptions" but also provided actors with powerful visual inspiration.

• Comparison with Existing Solutions:

  • Compared to tools like Zoom and Skype, Virtual Director significantly enhanced the immersion and interactivity of theatrical performances.
  • User feedback indicated that the tool effectively addressed issues of "Zoom fatigue" and attention dispersion.

• Experimental or Evaluation Results:

  • Over six weeks, 16 professional improvisational actors participated in online rehearsals and performances using Virtual Director.
  • All participants described positive experiences, including "fun," "excitement," and "creative stimulation."
  • Many participants reported feeling better able to interact with the virtual environment, even expressing that it allowed for greater physical expression than in-person interactions.

• Limitations and Future Directions:

  1. Technical Limitations: Issues such as video latency, device differences, and screen size still need to be addressed.
  2. Testing Condition Limitations: The study was conducted under isolation conditions, lacking analysis of diverse controlled environments.
  3. Future Directions:
     • Explore ways to optimize devices and operational environments to further minimize visual distractions.
     • Expand research to public performances involving live audiences, investigating audience integration and feedback mechanisms.
     • Design more automated features to simplify background switching and performance management, reducing technical barriers.

Conclusion

• The study demonstrates that "immersive video platforms" can significantly enhance the remote theatre performance experience, opening new avenues for remote theatrical creation during exceptional circumstances.
• By using virtual environments to enhance presence and interactivity in improvisational performances, this research provides valuable insights for the future development of other remote collaboration tools.