From Letterboards to Holograms: Advancing Assistive Technology for Nonspeaking Autistic Individuals with the HoloBoard

Title of the Paper

From Letterboards to Holograms: Advancing Assistive Technology for Nonspeaking Autistic Individuals with the HoloBoard

Paper Information

• Subject Area: Human-Computer Interaction, Augmented Reality (AR), Autism Communication Assistive Technology
• Keywords: Cross-Reality, Extended Reality, Nonspeaking Autism, Assistive Technology, Accessible Interaction, HoloBoard, Augmented Reality, Autonomous Communication, Sensory Regulation, Educational Technology

Research Background and Issues

• Identified Problems or Challenges:

  • Approximately one-third of autistic individuals cannot effectively communicate through spoken language (i.e., "nonspeaking").
  • Common augmentative and alternative communication (AAC) systems (e.g., picture sequence boards) are limited in vocabulary and expression, failing to support complex generative language.
  • Nonspeaking autistic individuals often face underestimated intellectual abilities, limited educational opportunities, and social isolation.
  • Current letterboard spelling methods for autistic individuals heavily rely on support personnel (Communication and Regulation Partner, CRP), increasing communication inconvenience and cost.

• Significance of the Research:

  • Developing independent, generative language communication systems for nonspeaking autistic individuals can significantly improve their education, social interaction, and quality of life.
  • Introducing novel technologies (e.g., augmented reality) may help autistic individuals better adapt and enhance autonomous communication capabilities, reducing dependency on support personnel.

• Motivation and Related Work:

  • The authors propose the "virtual letterboard" system (HoloBoard) using augmented reality (AR) devices, aiming to design a more independent and adaptive training environment based on nonspeaking individuals' existing letterboard spelling skills.
  • The study emphasizes the limited research on AR technology for autistic populations, with even fewer studies focusing on nonspeaking individuals.

Proposed Solution

• Proposed Solution:

  • Developed a virtual letterboard system (HoloBoard) based on HoloLens 2, simulating common physical letterboard scenarios.
  • Provided short training modules (average duration under 10 minutes) with progressively challenging spelling tasks, supporting a transition from simple letters to complete sentences.
  • The system allows support personnel (CRP) and users to share an augmented reality space, with added personalized interface settings.

• Innovative Aspects:

  • Applied augmented reality technology to the field of generative language communication, creating new possibilities for nonspeaking autistic individuals.
  • Leveraged users' existing spelling skills by designing the virtual letterboard interface and functionality to closely resemble physical letterboards, enhancing learning transfer.
  • Automated certain CRP tasks (e.g., letter recording, voice feedback), enabling users to interact with the system more independently.

• Implementation Steps and Key Technologies:

  • Core tools developed include the Unity game engine and the Microsoft Mixed Reality Toolkit (MRTK).
  • Implemented real-time shared virtual space synchronization for multiple users (based on Microsoft Azure Spatial Anchors technology).
  • Designed various user interface enhancements, such as auditory feedback, visual animations, and customizable colors when buttons are touched, compensating for the lack of physical tactile feedback in the virtual environment.

Research Outcomes

• Specific Achievements:

  • Among 23 participants, 16 completed the full training module tests, with an average training duration of just 9.66 minutes.
  • During testing, participants achieved an average spelling accuracy of 95.8%, demonstrating strong adaptability to the system.
  • Fourteen participants interacted freely, including spelling sentences and providing system feedback, while five participants independently entered "single-user mode" for autonomous spelling.

• Advantages Compared to Existing Solutions:

  • The system reduced the need for CRP physical intervention, achieving spelling training without CRP holding the board for the first time.
  • By combining virtual and real environments, the system improved user tolerance and independence.

• Experimental or Evaluation Results:

  • Participants' overall spelling speed (WPM) gradually improved, transitioning from 9.89 WPM on physical letterboards to 6.76 WPM on virtual boards (during the testing phase), indicating good adaptability to the system.
  • Users held positive attitudes toward the system's functionalities, recognizing its significant potential in education and private communication.

• Limitations and Future Directions:

  • Limitations:

    • Limited sample size (23 participants) consisting only of those familiar with physical letterboard spelling.
    • Short study duration, lacking long-term validation of the impact of virtual spelling.
    • The comfort of the device (HoloLens 2) remains a challenge for some users.

  • Future Directions:

    • Investigate how nonspeaking individuals without prior physical letterboard training adapt to virtual letterboards.
    • Design long-term, repeated training scenarios to explore the system's potential in real-life communication contexts (e.g., education, social interaction).
    • Consider personalized virtual environment customization to further integrate privacy and interactive communication while addressing device wearability issues.