FabSound: Audio-Tactile and Affective Fabric Experiences Through Mid-air Haptics

Title of the Paper

FabSound: Audio-Tactile and Affective Fabric Experiences Through Mid-air Haptics

Paper Information

• Field of Study: Human-Computer Interaction, Audio-Tactile Interaction, Digital Haptic Technology
• Keywords: Audio-tactile, Mid-air Haptic, Textile, Affective Haptics, Tactile experiences, Texture perception, Digital touch, Fabrics

Research Background and Problem

• Issues and Challenges:

  • The development of mid-air haptic technology enables tactile perception without physical contact, but accurately reproducing fabric textures using this technology remains a challenge.
  • Existing studies indicate that sound can influence the perception of tactile textures, but how audio-tactile integration can be used for simulating the tactile experience and emotional responses of digital fabrics is still unclear.

• Significance:

  • The tactile sensation of fabrics often influences consumer comfort evaluations and purchasing decisions. Reproducing realistic digital fabric experiences is significant in scenarios such as e-commerce and interaction design.
  • Integrating audio input into tactile experiences could provide new opportunities for creating more immersive and multi-sensory digital haptic interfaces.

• Research Motivation and Related Work:

  • Current research on multi-sensory roughness perception mainly focuses on physical materials, with limited exploration of such effects in digital haptic environments.
  • Studies on physical textile materials have established rich quantitative methods for tactile attributes, but mapping these textile characteristics to digital haptic systems remains underexplored.

Solution

• Proposed Method or Solution:

  • Utilize ultrasonic mid-air haptic technology to map real fabric textures onto mid-air haptic devices based on image texture information.
  • Combine mid-air haptic fabric textures with various audio signals (e.g., recorded fabric friction sounds and pure tones of different frequencies) to study their effects on tactile perception and emotional experiences.

• Innovations:

  • Integrate audio and haptics into the domain of digital fabric texture perception, further exploring the impact of audio-tactile interaction on tactile perception and emotional responses.
  • Validate changes in user experience under different perceptual modes through bidirectional experiments (single haptic vs. combined audio-tactile), addressing the research gap between cognition and behavior.

• Implementation Steps:

  1. Experimental Design:
     • In Experiment 1, select nine real cotton fabric samples, extract their displacement maps and normal maps, and convert them into mid-air haptic formats.
     • In Experiment 2, use digital suede fabric textures, overlaying recorded friction sounds and pure tones of different frequencies to study users' audio-tactile interaction experiences.
  2. Multimodal Mapping and Stimulus Generation:
     • Generate mid-air haptic materials using specific algorithms, while enhancing texture reproduction through microscopic image analysis.
  3. Evidence Collection and Analysis:
     • Employ a combination of quantitative questionnaires and semi-structured interviews to gather user data on texture roughness perception and emotional responses.

Research Outcomes

• Specific Findings:

  • Users found it difficult to directly match digital mid-air haptic textures with physical fabric samples but could discern roughness differences between various mid-air fabric textures.
  • Audio (e.g., high-frequency pure tones) significantly enhanced users' perception of roughness in mid-air digital fabric textures, while friction sounds were more likely to evoke fabric-related emotional experiences.
  • The combined audio-tactile perception elicited more emotional responses compared to standalone mid-air haptics. For instance, when experiencing soft and smooth textures (e.g., with added friction sounds), users were more likely to feel calm.

• Advantages and Disadvantages:

  • Compared to traditional direct tactile feedback, mid-air haptic technology combined with audio input significantly altered subjective perceptions of texture characteristics, offering stronger detail-shaping capabilities.
  • Compared to standalone visual or auditory content, audio-tactile combined experiences provide more immersive interaction scenarios and emotional connections in digital environments.

• Experimental and Evaluation Results:

  • The first experiment revealed that mid-air haptic technology tends to render fabric textures as smoother perceptions, especially when the original physical samples had higher roughness.
  • The second experiment confirmed that pure tones at 450Hz and 900Hz significantly enhanced texture roughness perception, while low-frequency pure tones and friction sounds were closer to the authentic tactile feel of physical textiles.

• Limitations and Future Directions:

  • Current mid-air haptic technology has limitations in reproducing the fine details of textures, making it difficult to fully match the complex tactile properties of fabrics.
  • The sample size of user experiments was relatively small; future studies could expand the sample group to enhance generalizability.
  • Further exploration of audio-tactile integration with temperature cues could extend multi-sensory design dimensions, enhancing the emotional impact of digital content.

Conclusion

• Main Conclusions: Mid-air haptic technology combined with audio input demonstrates unique potential in shaping the perception of roughness and emotional responses in digital fabric textures, providing important insights for future multi-sensory interaction design.

• Applications and Prospects:

  • Enhancing online fabric purchasing experiences in e-commerce.
  • Increasing user interaction immersion in public digital signage.
  • Optimizing design tools through open plugins that enable adjustable multi-sensory design solutions.