Augmenting Imagery with Multimodal Vibrotactile Representations: Touch, Feel, and Hear

Paper Title

Augmenting Imagery with Multimodal Vibrotactile Representations: Touch, Feel, and Hear

Publication Info

• Topic area: Accessibility and multimodal haptic feedback for blind or visually impaired users.
• Keywords: Vibrotactile feedback, material perception, blind users, accessibility, haptic feedback, AI-generated patterns, sensory substitution, tactile graphics, multimodal interaction, image accessibility.

Background and Problem

• Problem / challenge: Digital images are inaccessible to blind or visually impaired (BVI) users because alternative text and current accessibility tools fail to convey tactile and auditory material properties.
• Significance: Material qualities such as texture, hardness, and weight are critical for situational understanding, yet current tools focus on spatial and structural aspects, leaving a gap in material perception.
• Motivation and related work: Prior research in sensory substitution and haptics has explored tactile and auditory feedback for navigation and object recognition but has not addressed rendering material properties in images. Existing haptic databases and AI-driven methods for vibrotactile feedback have not been systematically evaluated for BVI users in this context.

Solution

• Proposed approach: Augment digital images with vibrotactile material representations using a multi-local vibrotactile tablet and four pattern generation methods: AI prompt-based synthesis, audio-to-pattern conversion, real material recordings, and a public haptic database.
• Novelty:
  1. Introduction of vibrotactile material feedback for image accessibility.
  2. Systematic comparison of four pattern generation pipelines for material perception.
  3. Empirical evaluation with BVI participants, integrating qualitative and quantitative insights.
  4. Preliminary design guidelines for material-specific vibrotactile feedback.
• Procedure and key techniques:
  1. Developed four pattern generation methods:
     • AP1: AI-generated patterns using one-shot learning.
     • AP2: AI-generated audio converted to vibrotactile patterns.
     • AP3: Real material recordings converted to vibrotactile signals.
     • AP4: Patterns from a public haptic database.
  2. Used a custom vibrotactile tablet with 16 piezoelectric actuators to deliver localized feedback.
  3. Conducted a user study with 8 BVI participants, who ranked patterns for 10 materials and provided think-aloud feedback.

Results

• Concrete findings:
  • Real_Recording (AP3) achieved the highest median rank (3/4) across materials, followed by HDB (AP4, median 2.3/4).
  • AI-generated patterns (AP1 and AP2) performed comparably to HDB, with specific strengths for certain materials (e.g., LLM_One-Shot for ceramic, LLM_Audio for metal and water).
  • Participants highlighted realism, distinctiveness, and personal associations as key factors influencing preferences.
• Advantage over baselines:
  • Real_Recording provided the most authentic patterns but required high-quality recordings.
  • AI methods offered flexibility and potential for automated generation but sometimes produced exaggerated or subtle patterns.
  • HDB patterns were effective for rough, rigid textures but lacked adaptability for softer or liquid materials.
• Experiments / evaluation:
  • Participants ranked patterns for 10 materials (e.g., wood, metal, water) and provided qualitative feedback.
  • Rankings were converted to Borda scores, and thematic analysis identified five key themes: realism, distinctiveness, personal associations, effort, and preferences.
• Limitations and future work:
  • Small sample size (n=8) limits generalizability.
  • Focused on ranking and qualitative feedback; did not measure identification accuracy or response times.
  • Future work should explore larger datasets, real-time material sensing, multimodal integration, and personalization.

Summary

This study introduces a novel approach to augmenting digital images with vibrotactile material feedback for blind and visually impaired users. Four pattern generation methods were evaluated using a custom vibrotactile tablet, with Real_Recording achieving the highest overall preference. AI-generated patterns showed promise but require further refinement for realism and distinctiveness. The findings highlight the potential of combining tactile and auditory cues to enhance image accessibility and provide actionable design guidelines for material-specific vibrotactile feedback. Future work will focus on scaling the system, integrating multimodal cues, and enabling real-time material sensing.