Exploring the Potential of Music Generative AI for Music-Making by Deaf and Hard of Hearing People

Research Background and Issues

• Identified Problems or Challenges
  The authors observed that while text-to-music generation AI technologies have shown significant potential in music education and therapy, these technologies largely exclude Deaf and Hard of Hearing (DHH) individuals. Current approaches typically focus only on basic musical elements (e.g., rhythm) and require assistance from hearing individuals for complex music creation, thereby limiting the independence of DHH users. Additionally, many DHH individuals have negative attitudes toward music and lack confidence in actively expressing themselves through music creation.

• Significance
  Music creation is an important art form that fosters multisensory interaction, learning, and self-expression. Providing DHH individuals with equal opportunities for music creation is an essential part of advancing cultural accessibility. From the perspective of inclusive art, enabling DHH individuals to independently create music through technology carries profound social and emotional significance.

• Research Motivation and Related Work
  Although some studies have enhanced DHH individuals' understanding of music through visual and tactile feedback, most of these studies focus on music appreciation or basic rhythm creation, with little exploration of independent, multidimensional music creation. Meanwhile, advancements in generative music AI technologies offer new possibilities: enhancing the convenience and complexity of music generation through natural language interaction.

Solution

• Proposed Method or Solution
  The authors designed and developed a multimodal music creation tool that integrates natural language input, visual, and tactile feedback. This system enables DHH users to independently create and edit music using generative music AI (GenAI) while providing intuitive alternative sensory feedback to understand musical content.

• Innovations

  • The tool employs generative music AI to create music from text input and translates it into visual and tactile information perceivable by DHH individuals.
  • The system supports music editing based on personal intent, combining lyric generation and music sequence visualization, allowing users to directly modify the musical output.
  • Design requirements tailored for DHH users are proposed, such as conveying emotions and musical atmosphere through visual elements (color, font, animation).

• Implementation Steps and Key Technologies

  1. Music Generation Module: Provides AI-based music generation using natural language input; supports both lyric-based and instrumental music creation.
  2. Multimodal Sensory Substitution Module: Translates generated music into visual (e.g., colors, animations, lyric associations) and tactile feedback (e.g., vibration intensity and rhythm) to enhance perception.
  3. Music Editing Functionality: Allows users to modify melody, emotion, rhythm, and other musical elements based on visual cues, optimizing the output with natural language prompts.
     Key technologies include a generative music AI framework (e.g., Suno), AI-supported music analysis APIs, and the integration design of tactile feedback devices.

Research Outcomes

• Specific Outcomes

  • Developed a multimodal music creation assistance tool that enables DHH users to independently create music.
  • The tool effectively improved DHH users' understanding of music through visual and tactile feedback, enhancing their ability to actively modify music.
  • DHH participants in the experiments expressed high satisfaction with the music creation process and outcomes, with many experiencing confidence and interest in music creation for the first time.

• Advantages Compared to Existing Solutions

  • Provides a complete experience from lyric generation to multidimensional music creation without relying on hearing individuals.
  • Unique sensory feedback design (visual + tactile) allows DHH users to actively evaluate and modify music.
  • Simultaneously promotes users' musical knowledge growth and interest in music during the creation process.

• Experimental or Evaluation Results

  • In tests with 9 DHH participants, the tool significantly improved users' independence and confidence in music creation (average satisfaction score of 6.5/7).
  • Participants demonstrated strong interest in expressing emotions and social connections through music, with some stating that the tool helped redefine their perception of hearing impairment.

• Limitations and Future Directions

  • Limitations: The current experiment has a small sample size, and the time delay in music generation (1-3 minutes) may weaken the coherence of the creation process.
  • Future Directions:
    • Enhance the real-time responsiveness and accuracy of the AI model to improve interaction experience.
    • Expand multimodal input support (e.g., gesture input, drawing interface) to better accommodate diverse expression methods among DHH users.
    • Add personalized feedback settings, allowing users to adjust feedback modes based on individual sensory preferences (e.g., choosing visual feedback only or stronger tactile cues).

By proposing and validating the integration of generative music AI with multimodal sensory substitution systems, this study provides a new pathway to enhance DHH individuals' participation in music creation and promote artistic inclusivity.