Keep the VRhythm Going: a Musician-Centred Study Investigating How Virtual Reality Can Support Creative Musical Practice

Title of the Paper

Keep the VRhythm going: A musician-centred study investigating how Virtual Reality can support creative musical practice.

Paper Information

• Subject Area: Virtual Reality and Musical Creative Practice
• Keywords: Virtual Reality, Music, Musical Practice, Rehearsal, Performance, Improvisation, Creativity

Research Background and Issues

• Problems or Challenges:

  • Musicians' creative practices are influenced by the acoustic and visual feedback of performance spaces, but rehearsals often cannot take place in the final performance venue.
  • Acoustic design in band rehearsal rooms is often of poor quality, failing to simulate complex perceptual effects.
  • The COVID-19 pandemic exacerbated difficulties in musicians' creative practices, such as the lack of physical spaces and the inability to transition into an appropriate mindset.
  • Research on Virtual Reality (VR) in performance has predominantly focused on audiences rather than performers, with existing systems lacking realistic sound rendering.

• Significance:

  • Enhancing musicians' adaptability to acoustic and visual environments can benefit creation, rehearsal, and performance.
  • Seeking solutions to provide alternative high-quality rehearsal spaces and alleviate performance anxiety.

• Research Motivation and Related Work:

  • Investigating how VR can support musicians' creative practices by designing musical experiences that integrate visual and acoustic elements.
  • Developing a VR system based on six design requirements and comparing it with existing simulators and spatial sound rendering studies.

Solution

• Methods and Solution:

  • Propose and develop a musician-centered VR system named "VR Rehearse & Perform."
  • Plan key design requirements based on feedback from musicians and performance science experts.
  • The system offers virtual environments with high-quality acoustic spaces and visual effects for rehearsal, performance, and improvisation.

• Innovations:

  • Enables musicians to explore customizable creative rehearsal and performance environments (acoustic properties and lighting effects).
  • Facilitates inspiration and the cultivation of performance mental states through interactions between acoustic spaces and visual environments.
  • Provides real-time sound feedback seamlessly integrated with virtual environments, while addressing audio quality and latency optimization.

• Implementation Steps and Techniques:

  • Build medium-fidelity and high-fidelity VR models using the Unity game engine.
  • Utilize the 3DTi toolkit for spatial audio processing, combined with real concert hall impulse responses (RIR).
  • Add dynamic interaction options, including gesture control and environment customization.
  • Collect data and evaluate the system's effectiveness in supporting musicians' creative practices.

Research Outcomes

• Specific Outcomes:

  • VR Rehearse & Perform effectively supports various creative practices of musicians, such as rehearsal, performance, and improvisation.
  • Provides access to high-quality acoustic environments and "sanctuary" spaces, enhancing performance preparation and mental states.
  • Explores the relationship between diverse musical environments and parameter settings with musical creativity.

• Advantages:

  • Offers an affordable, convenient, and flexible solution for musical rehearsal and creation.
  • The experimental environment inspires musicians' creativity and facilitates exploration of new performance possibilities.
  • Reduces the pressure of physical spaces and mitigates performance anxiety.

• Experimental or Evaluation Results:

  • Participating musicians generally recognized the potential of virtual environments and the flexibility of the system.
  • The acoustic accuracy of the system received positive feedback across different scenarios, though audio latency impacted the performance of certain instruments.
  • Environmental diversity significantly enhanced musicians' creativity and engagement.

• Limitations and Future Directions:

  • Limitations:
    • Audio latency issues were more pronounced for certain instruments, such as keyboards.
    • The VR system currently does not support multi-user interaction or real-time collaboration.
    • The system may lack comprehensive tracking of musicians' physical dynamics and instrument positioning.
  • Future Directions:
    • Improve audio processing techniques to reduce latency while maintaining acoustic authenticity.
    • Explore symbolic or multimodal hybrid environments to introduce new sound and visual interaction pathways for creative practices.
    • Develop virtual music platforms supporting multi-user collaboration and enhance visual integration of instruments.

Through this study, the authors successfully demonstrated that VR technology can serve as a powerful tool to support musical creative practices, laying a foundation for future related research.