A Way for Deaf and Hard of Hearing People to Enjoy Music by Exploring and Customizing Cross-modal Music Concepts

Title of the Paper

A Way for Deaf and Hard of Hearing People to Enjoy Music by Exploring and Customizing Cross-modal Music Concepts

Paper Information

• Research Domain: Accessibility Technology and Music Perception
• Keywords: Deaf and Hard of Hearing (DHH), Music Appreciation, Sensory Substitution Systems for Music, Conceptualization, Cross-modal Mapping

Research Background and Problem

• Problem or Challenge: Deaf and Hard of Hearing (DHH) individuals face difficulties experiencing music through traditional auditory media. Even with alternative systems using visual and tactile feedback, it remains challenging to fully understand and appreciate music. Many existing music substitution systems fail to significantly enhance the music appreciation experience, primarily due to the lack of intuitive and personalized cross-modal mapping design.
• Significance: Music is not only an art form but also a medium for rich emotional expression, social interaction, and personal intellectual fulfillment. Enhancing the natural perception and appreciation of music for DHH individuals can greatly improve their quality of life and opportunities for social engagement.
• Research Motivation and Related Work:
  • Literature review and focus group interviews identified existing issues in music substitution systems, including insufficient transmission of musical information and the lack of support for user adaptation during the design phase.
  • Exploring the potential of sensory substitution systems based on visualization and tactile feedback to address users' perceptual barriers to musical elements during music appreciation.

Solution

• Proposed Solution:
  • Developed a sensory substitution system for music based on a "cross-modal conceptualization framework," incorporating two core strategies: 1) Exploration, using virtual instruments to help users understand the mapping relationships between musical elements and substituted sensory modalities; 2) Personalized customization, enabling users to adjust sensory feedback to meet individual preferences.
• Innovative Contributions:
  • First-time proposal and validation of the necessity and effectiveness of the cross-modal conceptualization framework in music appreciation.
  • Introduction of personalized adjustment tools, such as options for visual elements (shape, color, size) and tactile stimuli (intensity and sensitivity), integrating user needs into system design.
• Implementation Steps and Key Technologies:
  • Conducted focus group interviews to identify pain points and needs of DHH individuals.
  • The system is divided into an exploration phase (learning modal relationships through simulated instruments) and a customization phase (adjusting visual and tactile feedback).
  • Experimental design involved comparative testing with 28 DHH participants to validate the enhancement of music appreciation experience through the cross-modal conceptualization framework.

Research Outcomes

• Specific Outcomes:
  • Experiments confirmed that the experimental group using the cross-modal conceptualization framework showed significant improvements in music appreciation experience, including enhanced perception of musical elements (pitch, loudness, and rhythm), increased satisfaction, and improved emotional resonance (pleasure and excitement).
  • The personalized customization phase significantly increased participants' interest in music, with some participants regaining a positive attitude toward music due to the customization capabilities.
• Comparative Advantages Over Existing Solutions:
  • Achieved highly personalized design of visual and tactile feedback, allowing users to freely adjust and express their individual music experiences.
  • Addressed the lack of intuitive cross-modal mapping in existing systems, improving system acceptance and practicality.
• Experimental and Evaluation Results:
  • In the experiment, the experimental group showed significant improvements in all dimensions of music appreciation (pitch, loudness, rhythm perception, satisfaction, emotions, etc.) by the fifth test, while the control group exhibited only slight improvements or even declines.
  • Qualitative analysis revealed that the exploration and customization phases helped participants gradually understand modal mapping, transforming music perception from monotonous, meaningless signals into recognizable, integrated musical experiences.
• Limitations and Future Directions:
  • Current modal mappings are relatively basic; future work could expand to include more musical elements (e.g., harmony and timbre).
  • The system currently uses only virtual piano and drum instruments; future research could explore more instruments and forms of musical expression.
  • Expanding the sample coverage is necessary to verify the system's applicability across different groups with hearing impairments.
  • On the technical platform, future development should focus on creating more portable hardware devices or mobile application versions.