From Daily Song to Daily Self: Supporting Emotional Growth of Deaf and Hard-of-Hearing Individuals through Generative AI Songwriting

Paper Title

From Daily Song to Daily Self: Supporting Emotional Growth of Deaf and Hard-of-Hearing Individuals through Generative AI Songwriting

Publication Info

• Topic area: Generative AI for emotional well-being and creative expression in Deaf and Hard-of-Hearing (DHH) individuals.
• Keywords: Generative AI, Deaf and Hard-of-Hearing, songwriting, emotional growth, self-expression, music therapy, conversational agents, accessibility, emotion regulation, journaling.

Background and Problem

• Problem / challenge: Existing music-based tools for DHH individuals focus on short-term benefits and lack support for sustained emotional reflection or iterative songwriting. DHH individuals often struggle to articulate emotions in lyrics or translate them into music, and prior systems do not address these challenges over multiple sessions.
• Significance: Sustained songwriting could provide deeper emotional benefits, such as self-insight and emotion regulation, for DHH individuals. This is particularly relevant given the psychosocial challenges (e.g., social isolation, stigma) faced by this group.
• Motivation and related work: Prior research has shown that songwriting can promote self-expression and identity reconstruction, but these studies have largely been limited to single-session interventions. Journaling and therapeutic songwriting suggest that repeated self-expression fosters emotional healing, but no prior work has explored long-term songwriting for DHH individuals using generative AI.

Solution

• Proposed approach: SoulNote, a generative AI-based system that integrates a conversational agent (CA) and music generation to enable DHH individuals to engage in iterative songwriting as a form of emotional journaling.
• Novelty:
  1. Multi-phase, user-centered system design involving music therapists and iterative feedback from DHH users.
  2. Development of a journaling-oriented system with features like a reflective CA, lyric-editing interface, multimodal music appreciation, and song archiving.
  3. In-the-wild validation of long-term emotional effects through a three-week diary study with DHH participants.
• Procedure and key techniques:
  1. A structured four-state songwriting process: music concept setting, lyric creation, music creation, and appreciation/discussion.
  2. Narrative-based and imagery-based questioning to guide users in expressing emotions and creating lyrics.
  3. Context-based music style recommendations to assist users in exploring diverse musical expressions.
  4. Integration of a web-based platform with GPT-4.1 for dialogue and a music generation API for creating personalized songs.

Results

• Concrete findings:
  • High user satisfaction with the songwriting process (M=6.04, SD=0.91) and musical outcomes (M=6.07, SD=1.01).
  • Average session duration: 41 minutes 24 seconds (SD=12m 56s).
  • Participants reported increased self-insight, emotion regulation, and shifts in everyday attitudes toward emotions and self-care.
• Advantage over baselines: Compared to text-only, CA-only, and music-only journaling conditions, SoulNote achieved the highest scores in self-awareness, emotional expression, and emotional change, demonstrating the value of integrating lyrics, dialogue, and music.
• Experiments / evaluation:
  • A three-week diary study with 12 DHH participants (12 sessions each) to evaluate long-term emotional and behavioral changes.
  • A follow-up study with 10 new participants comparing SoulNote to alternative journaling configurations.
  • Metrics included self-reports (e.g., satisfaction, emotional change) and behavioral logs (e.g., lyric editing time, replay counts).
• Limitations and future work:
  • Small and specific sample (DHH individuals with residual hearing); findings may not generalize to born Deaf individuals or sign-language users.
  • Single-arm study design limits causal inference; future work should include controlled comparisons and longer-term follow-ups.
  • Limited granularity in music generation; future systems should offer more controllable outputs tailored to individual hearing characteristics.

Summary

This study introduces SoulNote, a generative AI system designed to support sustained songwriting and emotional growth among DHH individuals. Through a structured, user-centered design process, the system integrates a conversational agent, music generation, and journaling features to enable iterative self-expression. A three-week diary study demonstrated that SoulNote fosters self-insight, emotion regulation, and positive behavioral shifts, with participants valuing the system as a resource for self-care and emotional resilience. While promising, future work should address broader inclusivity, causal evaluation, and enhanced music generation capabilities.