Social Sense-making with AI: Designing an Open-ended AI experience with a Blind Child

Title of the Paper

Social Sensemaking with AI: Designing an Open-ended AI Experience with a Blind Child

Paper Information

• Research Area: Artificial Intelligence and Human-Computer Interaction, Social Perception, Design for Visually Impaired and Blind Individuals
• Keywords: Artificial Intelligence, Human-Computer Interaction, Human-AI Collaboration, Intelligent Systems, Accessibility Design, Blind Perception Scanning, Visual Impairment, Children, Disability Design

Research Background and Problem

• Problem or Challenge:

  • Current AI technologies often focus on single tasks, such as facial recognition in images, with limited research exploring AI designs that support users in continuous, real-time perceptual experiences.
  • Blind and low-vision individuals face significant barriers in social integration and autonomous interaction, such as difficulty in understanding the positions or states of others in their surroundings.
  • Designing social perception AI systems for children is challenging, requiring concise and dynamically comprehensible information while addressing uncertainties in AI capabilities.

• Significance:

  • Blind children experience notable gaps in social interaction with peers, affecting their language development, emotional recognition, and educational outcomes.
  • Enhancing blind children’s self-awareness and interaction capabilities in social scenarios is critical for achieving educational equity and social inclusion.

• Research Motivation:

  • Existing technologies are often confined to discrete task designs rather than open-ended, real-time interactions.
  • Investigating how open-ended AI experiences can empower blind children’s social perception abilities to deepen their interaction with others.
  • Aiming to alleviate the information isolation faced by blind individuals in social contexts and enable more proactive and precise engagement with their surroundings.

Solution

• Method or Solution:

  • Proposed and designed a head-mounted augmented reality device named “PeopleLens,” integrating multi-algorithm AI real-time analysis to provide blind children with continuous, dynamic voice prompts about their social environment.
  • Created a real-time interactive AI experience by conveying spatial information about people’s positions and states through dynamic audio cues.

• Innovations:

  • Transitioning AI from “task completion” to “capability augmentation,” emphasizing collaborative value creation between humans and systems.
  • Low-information-density dynamic feedback design to help users focus on critical social information.
  • Combining spatialized audio output with external LED feedback designed for others, supporting the user’s social environment through AI outputs.

• Implementation Steps and Key Technologies:

  1. Hardware Device: Modified HoloLens device by removing visual display components, specifically designed for blind users.
  2. AI Model Pipeline: Utilized five computer vision algorithms to analyze the user’s social environment in real-time, tracking posture, facial recognition, gaze direction, and activity information within a four-meter radius.
  3. Audio Prompts:
     • Spatial audio cues to indicate the direction and identity of nearby individuals.
     • "Elastic audio" guiding users to adjust their head orientation to align with the target of focus.
  4. External LED Feedback: Added a light strip above the device to provide system recognition status to others during interactions.
  5. Scenario Testing:
     • Initial experiments conducted in laboratory settings, gradually transitioning to complex real-world environments such as schools and homes.

Research Outcomes

• Specific Results:

  • Designed and deployed PeopleLens, an open-ended AI system that dynamically supports blind children’s social perception.
  • Achieved low-information-density design, dynamic prompts facilitating mutual understanding between humans and AI, and external feedback mechanisms for interpersonal interaction transparency.

• Advantages:

  • Encouraged blind children to engage more actively in interactions through physical and attentional involvement, enhancing their social participation.
  • Integrated users’ existing perceptual abilities with AI system functionalities, forming a joint capability rather than merely replacing functions.
  • Enhanced participants’ “social presence” in social scenarios such as classrooms and family gatherings.

• Experimental or Evaluation Results:

  • Collected user feedback on the prototype system in real-world classroom and family settings:
    • TH (a 12-year-old blind child user) demonstrated noticeably more proactive physical orientation and attention behaviors.
    • Users showed improved adaptability in dynamically adjusting attention and locating others (e.g., subtle head movements enabling multiple system outputs).
  • Experiments revealed dynamic co-shaping between users and the AI system due to algorithmic diversity and environmental complexity.

• Limitations and Future Directions:

  • Limited sample size (focused on one participant), requiring larger-scale evaluation and validation.
  • The system currently lacks training for social skills and evaluation in complex interaction scenarios (e.g., large gatherings).
  • Opportunities to further optimize algorithm performance and device miniaturization.
  • Need to explore how to generalize this AI design paradigm and investigate collaborative mechanisms for different roles in social environments (e.g., teachers or parents).

Conclusion

This study opens new avenues for open-ended human-AI interaction design, demonstrating how AI can augment individual social capabilities. The design experience of PeopleLens provides valuable insights for intelligent accessibility technologies and lays a solid foundation for future designs of more complex perception-interaction systems.