Wish You Were Here: Mental and Physiological Effects of Remote Music Collaboration in Mixed Reality

Document Title

Wish You Were Here: Mental and Physiological Effects of Remote Music Collaboration in Mixed Reality

Document Information

• Subject Area: Human-Computer Interaction, Mixed Reality, Remote Music Collaboration
• Keywords: Mixed Reality, Augmented Reality, Remote Collaboration, Head-Mounted Display, Networked Music Performance, Co-presence, Social Presence, Physiological Signal Processing, Psychophysiology

Research Background and Problem

• Problem or Challenge:

  • Due to pandemic restrictions, face-to-face music collaboration has been severely impacted, making it difficult for musicians to experience "co-presence" with their partners.
  • Current remote music collaboration tools often neglect visual interaction, resulting in low social presence and limiting the non-auditory transmission of knowledge during music creation.
  • Although mixed reality technology has the potential to enhance "co-presence" and music creation experiences, related research and the technical foundation for high-quality real-time low-latency transmission remain underdeveloped.

• Significance:

  • Remote music collaboration is particularly important during the pandemic and in contexts of social isolation.
  • Enhancing co-presence and fostering "flow" states can promote individual mental health and motivation for completing music creation.

• Research Motivation and Related Work:

  • Music collaboration can improve self-esteem, emotional states, and mental health, but it also relies on emotional connection and non-verbal communication during face-to-face interactions.
  • Literature indicates that transmitting visual information (e.g., posture and gestures) can effectively enhance social presence in remote music collaboration, but the specific implementation and impact under mixed reality conditions remain to be explored.

Solution

• Method or Solution:

  • Propose a mixed reality-based remote music collaboration system that uses head-mounted displays (HMDs) to present real-time 3D point clouds, enhancing remote "co-presence."
  • Conduct experimental studies to examine the effects of "Mixed Reality" (MR) versus "Audio-only" (AO) conditions on musicians' flow, co-presence, and positive emotions.

• Innovation:

  • Utilize head-mounted displays and 3D point cloud capture technology to reconstruct the visual representation of remote musicians in real time.
  • Systematically integrate physiological data (e.g., heart rate variability, skin conductance activity) to analyze the physical characteristics of "flow" states.
  • Identify potential physiological features for real-time evaluation of "flow states," providing new directions for affective computing development.

• Implementation Steps:

  1. Use a prototype system to test musicians' subjective and physiological responses to co-presence, flow, and positive emotions under two conditions (MR and AO).
  2. Collect subjective feedback through questionnaires using standardized tools (e.g., NMSPI, FSS, PANAS) to assess psychological indicators.
  3. Gather physiological features (e.g., HRV and EDA) and analyze feature correlations with self-annotated "flow" periods.
  4. Conduct open-ended feedback for quality analysis, summarizing user suggestions and improvement recommendations for the system.

Research Outcomes

• Specific Findings:

  • Compared to "Audio-only," the MR condition significantly enhanced musicians' sense of co-presence (co-presence scale showed significant improvement, p=0.011).
  • Both conditions significantly increased musicians' positive emotions, but there was no significant difference between MR and AO in their effect on emotional enhancement.
  • Physiological data analysis identified heart rate (BPM) and low-frequency/high-frequency ratio (LF/HF) as significant predictors of "flow states," especially under MR conditions.
  • The "high heart rate-low LF/HF" feature of flow states was more pronounced in MR scenarios, indicating that MR may increase physiological arousal and emotional engagement.

• Advantages:

  • Unlike existing audio-focused solutions, MR technology supports multimodal integration of visual and non-verbal signals, thereby increasing social presence in music collaboration.
  • Real-time point cloud visualization under MR conditions makes remote collaboration more immersive, providing references for designing new tools to enhance musician interaction.

• Experimental or Evaluation Results:

  • Quantitative evaluations showed no significant difference in "flow scores" between the two conditions; qualitative feedback indicated that some users felt AO reduced visual distractions and enhanced musical focus.
  • Physiological data analysis revealed that "flow states" and "non-flow states" were better distinguished under MR conditions, supporting future research on affective computing based on real-time physiological signals.

• Limitations and Future Directions:

  • Limitations:
    • The current HMD system has limited resolution, and latency issues affected some musicians' precise control of instruments.
    • Point cloud quality and network latency for long-distance real-world applications were not fully addressed in the experiment.
    • Small sample size limited the statistical power of some indicators.
  • Future Directions:
    • Explore new hardware with higher resolution or lower latency (e.g., Varjo XR-3 or Microsoft HoloLens).
    • Develop mixed reality-assisted systems based on physiological signal detection of real-time flow states, dynamically adjusting challenges or support for musicians with varying skill levels.
    • Expand experimental scenarios to long-distance internet transmission to observe MR collaboration effects under different latency conditions.
    • Compare other display methods (e.g., 3D projection or screens) in enhancing co-presence and interaction.