Designing a Generative AI-Assisted Music Psychotherapy Tool for Deaf and Hard-of-Hearing Individuals

Paper Title

Designing a Generative AI-Assisted Music Psychotherapy Tool for Deaf and Hard-of-Hearing Individuals

Publication Info

• Topic area: Generative AI applications in inclusive music psychotherapy for DHH individuals.
• Keywords: Generative AI, music psychotherapy, Deaf and Hard-of-Hearing (DHH), songwriting, conversational agents (CAs), emotional expression, music therapy, inclusivity, multimodal interfaces, therapeutic design.

Background and Problem

• Problem / challenge: Traditional music psychotherapy is largely inaccessible to DHH individuals due to its auditory-centric nature. Existing interventions for DHH populations focus on auditory perception rather than emotional well-being, leaving a gap in inclusive therapeutic tools.
• Significance: Addressing this gap can provide DHH individuals with opportunities for emotional expression, healing, and self-awareness through music, fostering mental health and social inclusion.
• Motivation and related work: Prior research highlights the multisensory engagement of DHH individuals with music and the potential of generative AI (GenAI) in creative and therapeutic contexts. However, there is a lack of intervention studies that integrate GenAI to make music psychotherapy accessible and emotionally meaningful for DHH individuals.

Solution

• Proposed approach: A GenAI-assisted music psychotherapy tool combining conversational agents (CAs), music generation AI, and music visualization to enable collaborative songwriting for DHH individuals.
• Novelty:
  1. Co-designed with music therapists to tailor the tool to the sensory and emotional needs of DHH users.
  2. Incorporates three conversational strategies: supportive empathy, example response options, and visual-based metaphors.
  3. Provides a structured four-stage therapeutic process: therapeutic connection, making lyrics, making music, and song discussion.
  4. Integrates multimodal feedback and music visualization to enhance accessibility and emotional engagement.
• Procedure and key techniques:
  1. Conducted interviews with six music therapists to identify challenges in DHH music psychotherapy.
  2. Held a co-design workshop with four therapists to develop design principles and therapeutic processes.
  3. Developed a React-based web tool with a CA interface for songwriting and a music visualization interface.
  4. Implemented a state-step-based prompting framework for the CA to guide users through songwriting and emotional reflection.

Results

• Concrete findings:
  • Participants spent an average of 40.15 minutes using the tool, with high self-disclosure (M=6.02/7) and satisfaction with songwriting (M=5.86/7).
  • Emotional openness was facilitated by the CA’s empathetic responses, visual-based metaphors, and example options.
  • Emotional transformation was observed, with participants reporting catharsis, emotional reframing, and increased self-awareness.
• Advantage over baselines:
  • Enabled DHH individuals to engage in songwriting and emotional expression independently, overcoming barriers of traditional auditory-based therapy.
  • Provided a judgment-free, text-based interaction space that increased comfort and trust.
• Experiments / evaluation:
  • Conducted a usage study with 23 Korean CI users who identify as DHH.
  • Evaluated emotional openness, creative engagement, and usability through system logs, questionnaires, and post-interviews.
  • Observed therapeutic effects such as emotional release, objectification, and reflective dialogue.
• Limitations and future work:
  • Limited to Korean CI users with text-based communication; generalizability to other DHH populations is uncertain.
  • Emotional mismatches in generated music highlight the need for improved controllability in GenAI models.
  • Single-session evaluation; longitudinal studies are needed to assess sustained therapeutic outcomes.
  • Future iterations should involve DHH users in the co-design process and expand to diverse cultural and linguistic contexts.

Summary

This study presents a GenAI-assisted music psychotherapy tool co-designed with music therapists to support emotional expression and healing for DHH individuals. The tool employs conversational agents, music generation AI, and music visualization to enable collaborative songwriting, addressing the sensory and emotional needs of DHH users. A usage study with 23 participants demonstrated high satisfaction, emotional openness, and therapeutic potential, with users experiencing emotional transformation and self-reflection. While promising, the tool requires further development to enhance model controllability, involve diverse DHH populations, and validate long-term therapeutic efficacy. This work contributes to inclusive AI design and expands the possibilities for accessible mental health technologies.