EmojiFan: Designing A Social Interface Supporting Facial Expression Interaction for Blind and Low Vision People in Party Settings

Paper Title

EmojiFan: Designing A Social Interface Supporting Facial Expression Interaction for Blind and Low Vision People in Party Settings

Publication Info

• Topic area: Assistive technology for social interaction in accessibility contexts.
• Keywords: Blind and low vision, facial expression interaction, AI proxy, social assistive technology, party settings, emoji-based communication, haptic feedback, accessibility design, user study, human–AI interaction.

Background and Problem

• Problem / challenge: Blind and low vision (BLV) individuals face barriers to engaging in facial expression interactions in dynamic social environments like parties, where nonverbal cues are crucial. Existing assistive technologies focus on perceiving social cues but neglect support for responding to these cues.
• Significance: Social gatherings are key for interpersonal relationships, yet BLV individuals experience reduced participation, contributing to loneliness and social isolation. Addressing these barriers could enhance their autonomy and social inclusion.
• Motivation and related work: Previous systems like Google Glass and VR headsets assist BLV users in perceiving nonverbal cues but fail to support full facial expression interaction. Eye-tracking devices aid eye contact but overlook broader facial expressions. This paper aims to bridge these gaps, focusing on party settings.

Solution

• Proposed approach: EmojiFan, an AI-powered smart fan that translates facial expressions into dynamic emojis, enabling BLV users to perceive and respond to social cues in party settings.
• Novelty:
  1. Development of EmojiFan, an AI-assisted system for emoji-based facial expression interaction tailored for BLV users.
  2. Investigation of BLV users’ challenges in facial expression interaction after perceiving social cues.
  3. Design insights for creating socially acceptable, unobtrusive assistive technologies.
• Procedure and key techniques:
  • Conducted formative study with 10 BLV users to identify challenges in party settings.
  • Designed EmojiFan with features like haptic feedback, personalized emoji responses, and a fan interface for unobtrusive interaction.
  • Implemented hardware (ESP32-CAM, vibration motors, gyroscope) and software (fine-tuned GPT-4o, DeepFace model) for real-time facial expression recognition and emoji display.
  • Evaluated EmojiFan in a simulated party environment with 6 BLV users and 8 sighted partners.

Results

• Concrete findings:
  • EmojiFan facilitated ice-breaking and enhanced interaction for BLV users, making conversations smoother and more engaging.
  • BLV users gained autonomy in initiating and controlling social interactions, using the fan to signal engagement or disengagement.
  • Sighted individuals showed increased empathy and attentiveness toward BLV users, improving mutual understanding.
• Advantage over baselines:
  • Enabled richer facial expression interaction beyond eye contact, addressing gaps in prior assistive technologies.
  • Provided unobtrusive, culturally appropriate interaction through the fan interface, reducing stigma.
• Experiments / evaluation:
  • Simulated party setting with controlled interactions both with and without EmojiFan.
  • Data collected via video recordings, interviews, and thematic analysis.
  • Results showed higher perceived conversation quality when EmojiFan was used.
• Limitations and future work:
  • BLV users could not perceive the emoji displayed in real time, highlighting the need for alternative feedback mechanisms.
  • Ambiguities in AI-generated emojis and biases in facial recognition require further optimization.
  • Future studies should test EmojiFan in real-world party scenarios and explore its applicability in other social contexts.

Summary

This paper introduces EmojiFan, an AI-powered smart fan designed to support BLV individuals in facial expression interaction during parties. By translating facial expressions into dynamic emojis, EmojiFan enables BLV users to perceive and respond to social cues, fostering autonomy and reducing social stigma. User studies demonstrate its effectiveness in facilitating ice-breaking, enhancing empathy among sighted individuals, and improving the quality of interactions. Future work will address limitations in real-time feedback and explore broader applications in diverse social settings.