Seeing the Sound: Supporting Musical Collaboration with Augmented Reality

In musical collaboration, digital musical instruments often hinder effective communication and engagement by restricting visibility and limiting gestural and non-verbal interactions. These challenges reduce musicians’ situational awareness and complicate cohesive performance. To address this, we developed a head-mounted augmented reality (AR) system to enhance collaborative musical experiences by visualising musicians’ hand movements, eye gaze positions, and instrument interactions in real-time. We conducted a user study involving four pairs of musicians performing live music using different AR interface configurations. The results suggest that the AR system can enhance situational awareness and assist collaboration, as reflected in questionnaire responses. Interviews indicated that real-time visualisations of bodily movements and interactions helped participants better understand the collaborative process and anticipate their collaborators’ actions. These findings point to the potential of AR-assisted visualisation to support creative collaboration by tailoring visual information to different needs. Future research could explore its application in broader contexts of real-time creative cooperation.