Jess+: AI and robotics with inclusive music-making

Title of the Paper

Jess+: Research on AI and Robot-Assisted Inclusive Music Creation

Bibliographic Information

• Subject Area: Application of Artificial Intelligence and Robotics in Music Creation
• Keywords: Inclusive Design, Artificial Intelligence, Neural Networks, Robotics, Music Creation, Creative Collaboration, Digital Score

Research Background and Problem Statement

• Identified Problems or Challenges:
  1. Certain forms of music creation pose significant participation barriers for individuals with physical disabilities, such as limited physical flexibility or inability to operate traditional instruments.
  2. How to enhance the inclusivity of the creative process through technological means while introducing non-traditional creative triggers to ensembles remains an unresolved issue.
• Research Importance:
  1. Creating an inclusive music creation environment can provide equitable participation opportunities for musicians with physical disabilities.
  2. Exploring new creative tools and technologies for the music industry can drive further innovation in traditional scores and creative practices.
• Research Motivation and Related Work:
  1. This study is inspired by the "community ensemble" concept, including inclusive ensemble projects composed of musicians with and without disabilities (e.g., the Able Orchestra in the UK).
  2. The study centers around the concept of digital scores, examining their impact on musicality and creativity through technological transformation.
  3. The research integrates existing studies on artificial intelligence, robotic music creation, and neural networks while investigating collaborative models in inclusive design.

Solution

• Proposed Solution: The research team developed an AI robotic system named Jess+, designed around digital scores and collaborating with musicians with and without physical disabilities for interdisciplinary exploration. The system aims to enhance inclusivity and creativity in the music creation environment through AI and robotics.

• Innovative Features:

  1. Unlike traditional music-assistive tools, Jess+ combines artificial intelligence and robotics to interact with musicians through real-time "graphical scores" and spatial motion gestures.
  2. The system captures user input via physiological signals such as brainwaves (EEG) and electrodermal activity (EDA), forming a real-time feedback loop.
  3. By adopting a "human-machine co-creation" approach, the system fosters accessible and highly collaborative creative interactions.

• Implementation Steps and Key Technologies:

  1. System Architecture Design:
     • The system is based on an AI factory module, integrating seven neural network models (e.g., audio envelope, brainwave prediction) to generate interactive "response intensity" streams and drive robotic actions.
  2. Closed-Loop Interaction Model:
     • The system collects audio and physiological data in real-time, processes them through neural networks, and generates gesture control parameters (e.g., speed, acceleration).
     • The robot establishes a creative relationship with musicians through physical movements (drawing shapes or performing dance gestures).
  3. Collaboration and Iterative Improvement:
     • Over a four-month iterative workshop, the research team collaborated with three musicians to refine the Jess+ system, including debugging robotic movements and enhancing interaction logic.
     • Various experimental scenarios were designed around tactile AI and audio response to test the system's support for creative output.

Research Outcomes

• Specific Outcomes:

  1. The system provided a new creative platform for the physically disabled musician (Jess), enabling her to express emotions through physiological sensors and AI.
  2. Non-disabled team members regarded Jess+ as a "creative accompanist," enhancing their confidence and freedom in improvisation.
  3. The system effectively promoted inclusivity and collaboration in the music performance environment.

• Advantages:

  1. The system was perceived as "non-judgmental" and "non-directive," encouraging participants to take creative risks.
  2. Compared to traditional score creation tools, the system stood out for its interactive and dynamically generated actions, increasing randomness and inspiration triggers in the creative process.

• Experimental and Evaluation Results:

  1. Musicians felt that the robot was "listening" to their music and created a non-verbal connection during interaction.
  2. During experiments, performers reported that the system stimulated their creativity, fostering a mindset of musical risk-taking and confidence.
  3. When the system was perceived as an "ensemble member" like others, its participation brought richer-than-expected creative collaboration experiences.

• Limitations and Future Directions:

  • Limitations:
    1. Dependence on audio/physiological data may lead to user experience issues due to sensor malfunctions.
    2. The current system is limited to specific high-level musicians, with uncertain adaptability for beginners or diverse user groups.
  • Future Directions:
    1. Optimize interaction recognition mechanisms to enhance adaptability for broader user groups.
    2. Explore richer robotic expression methods (e.g., dance or non-musical expressions) to expand application scenarios.
    3. Design enhanced system iterations for educational institutions and inclusive social projects.

Conclusion

This study explores the potential of integrating AI and robotics into the fields of musical inclusivity and creativity. Jess+ is not merely a technological tool but a multifaceted bridge for music creation, inspiring creators to transcend their limitations and experience the liberating power of artistic innovation. It sets an important example for the comprehensive application of human-machine collaboration in the arts.