Plug the Guitar In Before You Put the Headset On: Co-designing a Mixed Reality Collaborative Application with Professional Musicians

Paper Title

Plug the Guitar In Before You Put the Headset On: Co-designing a Mixed Reality Collaborative Application with Professional Musicians

Publication Info

• Topic area: Mixed reality (MR) applications for professional musical collaboration
• Keywords: Mixed reality, networked music performance, co-design, professional musicians, audio-first design, non-verbal communication, spatial customization, collaborative systems, immersive technologies, music technology

Background and Problem

• Problem / challenge: Current MR systems prioritize visual immersion and are designed for office-based collaboration, failing to meet the unique requirements of professional musical collaboration, such as real-time synchronization, high-fidelity audio, and non-verbal communication.
• Significance: Addressing these gaps is crucial for enabling geographically distributed musicians to collaborate effectively, preserving the creative and professional standards of in-person musical collaboration.
• Motivation and related work: While networked music performance (NMP) systems like JackTrip and LoLa have achieved ultra-low latency audio transmission, they lack immersive and social dimensions. MR technologies offer potential solutions, but existing research focuses on technical feasibility or small-scale evaluations without systematically addressing professional musicians' needs.

Solution

• Proposed approach: A co-design methodology involving 24 professional musicians to identify design requirements for MR musical collaboration systems.
• Novelty:
  1. Empirically grounded design requirements for MR musical collaboration systems based on professional musicians' input.
  2. A three-phase co-design methodology combining hands-on MR experience, speculative design, and consensus-building.
  3. Evidence that professional creative collaboration demands different priorities than office-based MR applications.
• Procedure and key techniques:
  1. Phase 1: Experience grounding with MR technology to familiarize musicians with HMDs and headphones during collaborative performance.
  2. Phase 2: Individual ideation and sketching to envision ideal MR systems unconstrained by current limitations.
  3. Phase 3: Collaborative convergence to negotiate and prioritize shared design requirements.

Results

• Concrete findings:
  • Five unanimous requirements: audio-first workflow, plug-and-play simplicity, personalized audio control, spatial layout customization, and support for non-verbal communication.
  • Near-unanimous requirements: real-time voice support, stereo mix priority.
  • High-consensus requirements: realistic avatar representation, presets for configurations, departure coordination, and recording/playback capabilities.
• Advantage over baselines: Challenges the visual-first paradigm of MR systems by prioritizing audio and non-verbal communication, aligning with professional musicians' workflows and creative needs.
• Experiments / evaluation:
  • Participants: 24 professional musicians (average 12.7 years of experience) organized into six four-piece bands.
  • Methodology: Three-phase co-design process involving MR hands-on experience, speculative design, and group consensus.
  • Data collection: Audio recordings, sketches, and thematic analysis of group discussions.
• Limitations and future work:
  • Limited generalization to larger ensembles, non-Western musical traditions, and musicians with disabilities.
  • Technology constraints in Phase 1 (e.g., HMD-specific issues, co-located setup).
  • Future work should include distributed MR systems, diverse participant demographics, and implementation/testing of identified requirements.

Summary

This study identifies seven core design requirements for MR musical collaboration systems through a co-design process with professional musicians. Key findings include the need for audio-first workflows, non-verbal communication support, personalized audio and spatial controls, and plug-and-play simplicity. The results challenge visual-first MR paradigms, emphasizing the primacy of audio and established professional workflows. These requirements provide a foundation for developing MR systems that enable geographically distributed musicians to collaborate as if sharing physical space, preserving the creative and professional standards of in-person collaboration.