Harmonizing the Senses: Designing a Cross-Modal Interactive Art System to Enhance Older Adults’ Affective Experiences

Paper Title

Harmonizing the Senses: Designing a Cross-Modal Interactive Art System to Enhance Older Adults’ Affective Experiences

Publication Info

• Topic area: Cross-modal interactive systems for older adults’ affective and cognitive well-being.
• Keywords: Multisensory stimulation, cross-modal correspondence, affective congruency, AI-infused art systems, older adults, digital art, cognitive abilities, user experience, emotional well-being, human-computer interaction.

Background and Problem

• Problem / challenge: Existing multisensory stimulation (MSS) systems often lack scalability, adaptability, and emotional coherence. Poor synchronization across sensory modalities can lead to cognitive overload, disengagement, or negative affective experiences, particularly for older adults with cognitive and sensory declines.
• Significance: Enhancing older adults’ affective experiences through MSS can improve emotional well-being, cognitive engagement, and quality of life, addressing challenges posed by aging populations.
• Motivation and related work: Prior research highlights the benefits of MSS and digital art systems for older adults but lacks focus on cross-modal congruency and AI integration. Existing studies often prioritize sensory activation over emotional alignment, leaving gaps in designing inclusive, adaptive systems for older users.

Solution

• Proposed approach: Development of a cross-modal interactive art system incorporating AI to enhance older adults’ affective experiences by aligning visual, auditory, and kinetic modalities.
• Novelty:
  1. Empirical evidence on how cross-modal correspondence and congruency influence older adults’ affective responses.
  2. Design and evaluation of an AI-infused cross-modal art system that reduces cognitive load and enhances emotional engagement.
  3. Introduction of adaptive AI scaffolding to support users with varying cognitive abilities.
  4. Design guidelines for inclusive, multisensory systems tailored to older adults.
• Procedure and key techniques:
  1. Experiment I: Investigated timbre–color associations using traditional Chinese instruments and color cards, focusing on affective congruency.
  2. Experiment II: Explored auditory–kinetics synchrony in a digital painting system, testing five levels of sensory mapping and their effects on affective experiences.
  3. AI-infused system: Integrated AI to transform users’ strokes into dynamic ink-style artworks, synchronizing visual, auditory, and kinetic inputs for real-time feedback.

Results

• Concrete findings:
  • Timbre–color associations were stronger when affectively aligned, with brighter, sharper timbres linked to saturated, lighter, and red–yellow colors.
  • High auditory–kinetic synchrony elicited mixed responses; participants with lower cognitive abilities reported higher pleasure under certain conditions.
  • AI-infused modes significantly increased positive affect (pleasure, surprise, valence, arousal) and reduced negative affect (sadness, anger), particularly for users with lower cognitive abilities.
• Advantage over baselines:
  • AI-infused modes outperformed non-AI modes in enhancing emotional engagement and reducing cognitive load.
  • Participants with lower cognitive abilities experienced compensatory benefits, achieving affective states comparable to or better than higher-ability peers.
• Experiments / evaluation:
  • Experiment I: 35 participants (65–79 years) evaluated timbre–color associations using audio clips, color cards, and emotion measurement tools.
  • Experiment II: 32 participants tested five auditory–kinetics synchrony conditions in a digital painting system.
  • AI-infused system: Same 32 participants compared AI and non-AI modes across sensory integration conditions, with data collected via self-reports and facial expression analysis.
• Limitations and future work:
  • Limited sample diversity and ecological validity due to controlled experimental settings.
  • Focused only on auditory–visual and auditory–kinetic mappings; future studies should explore additional sensory modalities.
  • Need for longitudinal studies to assess sustained engagement and adaptive AI support over time.

Summary

This study developed and evaluated a cross-modal interactive art system for older adults, integrating AI to enhance affective experiences. Experiment I demonstrated that affective congruency strengthens timbre–color associations, while Experiment II revealed mixed responses to auditory–kinetic synchrony, with cognitive ability influencing preferences. The AI-infused mode significantly improved positive affect and reduced negative affect, particularly for users with lower cognitive abilities. These findings highlight the importance of affective congruency, adaptive AI scaffolding, and inclusive design in creating emotionally engaging and cognitively accessible multisensory systems for older adults.