Designing AACs for People with Aphasia Dining in Restaurants

Title of the Paper

Designing AACs for People with Aphasia Dining in Restaurants

Paper Information

• Subject Area: Design and application of Augmentative and Alternative Communication (AAC) devices to support social and leisure activities for people with aphasia
• Keywords: Aphasia, Augmentative and Alternative Communication (AAC), Accessibility Technology, Artificial Intelligence, Social Interaction, Language Impairment, Health Computing, User Experience, Usability Evaluation, Image Recognition

Research Background and Problem Statement

• Identified Problems or Challenges

  • Current AAC applications primarily focus on meeting basic functional needs (e.g., daily communication) but lack support for leisure activities such as ordering food in restaurants.
  • People with aphasia face multiple challenges when ordering food in restaurants, including the time-sensitive nature of language and the complexity of menu content.
  • Restaurant interactions require specific vocabulary, but existing AAC devices cannot quickly adjust vocabulary to suit new social contexts.
  • Ineffective communication may lead to frustration in social participation for people with aphasia, negatively impacting their quality of life.

• Significance

  • Language impairments can lead to social isolation, depression, and reduced quality of life for people with aphasia, especially in leisure settings like restaurants.
  • Effective communication tools that support leisure activities can enhance patients' autonomy and promote their social participation.

• Research Motivation and Related Work

  • The authors cite related literature highlighting the close relationship between the quality of life for people with aphasia and their social participation, with dining in restaurants being a goal-oriented yet challenging scenario.
  • Previous studies have explored how AAC technologies can improve language retrieval speed through predictive models or optimize vocabulary matching through context-aware technologies, but these methods still fall short in supporting dynamic language needs.

Proposed Solution

• Method or Solution

  • Through clinical observation, interviews, and prototype design, the authors propose three AAC prototypes (PhotoSearch, MenuSpeak, and OrderEat) to help people with aphasia independently order food in restaurants.
  • These prototypes integrate artificial intelligence technologies, including photo caption generation, OCR (Optical Character Recognition), geolocation data, and restaurant menu information retrieval.

• Innovative Aspects

  • Application of AI technologies (e.g., image captioning, OCR, and geolocation data) to AAC devices to address the challenges of specific language support needs in unfamiliar dining environments for people with aphasia.
  • The design emphasizes providing users with the autonomy to select language options rather than fully automating the process, supporting patients' decision-making rights.
  • The prototypes offer multimodal information combining text, images, and speech to facilitate language comprehension and communication.

• Implementation Steps and Key Technologies

  1. PhotoSearch:
     • Users take photos, and AI automatically generates captions for the images, linking them to related pictures to support vocabulary understanding.
     • Users can click on captions to generate speech output or expand related images.
  2. MenuSpeak:
     • OCR technology converts printed menus into interactive digital text, allowing users to click on words for speech playback and image display.
     • Provides ingredient selection functionality to help users customize their meals.
  3. OrderEat:
     • Real-time extraction of restaurant menu information using geolocation data.
     • Presents menus with a combination of images and text to support users in exploring and managing their orders.

Research Outcomes

• Specific Results

  • Designed and tested three prototypes to help people with aphasia overcome language barriers when ordering food in restaurants from different perspectives.
  • PhotoSearch supports users in acquiring new vocabulary through photo recognition, MenuSpeak offers real-time translation and interaction with printed menus, and OrderEat integrates geolocation data and menu information to support restaurant exploration and dining decisions.

• Advantages

  • Enhanced the autonomy and language production capabilities of people with aphasia in unfamiliar restaurant environments.
  • Provided tools for new language expression, addressing the limitations of traditional AAC devices in accommodating non-standard vocabulary needs.
  • Combined multimodal information (images, text, speech) to promote users' understanding of language.

• Experimental or Evaluation Results

  • Experiments showed that PhotoSearch's image diversity improved vocabulary comprehension, MenuSpeak's interactive text reduced the time needed to access words and phrases, and OrderEat supported users in exploring menu options more deeply, effectively reducing communication stress.
  • Participants generally reported that these tools improved their dining experiences and language communication abilities.
  • User feedback highlighted the reliability of photo capture and AI results as key areas for further optimization.

• Limitations and Future Directions

  • Limitations:
    • The accuracy of AI predictions still requires manual verification through multimodal information, especially as language comprehension challenges in people with aphasia amplify the issue.
    • Some prototypes (e.g., MenuSpeak and OrderEat) may not be user-friendly for individuals with severe aphasia, as tasks require certain visual or clicking capabilities.
    • The system still relies on users' upper limb control or visual perception abilities.
  • Future Directions:
    • Explore more advanced AI technologies to improve prediction accuracy.
    • Optimize interaction design to reduce cognitive load during use.
    • Expand the scope of support to other scenarios, such as retail points or public spaces for interactive communication.