“The less I type, the better”: How AI Language Models can Enhance or Impede Communication for AAC Users

Document Title

“The less I type, the better”: How AI Language Models can Enhance or Impede Communication for AAC Users

Document Information

• Subject Area: Research on the integration and application of Augmentative and Alternative Communication (AAC) technologies with Artificial Intelligence Language Models (LLMs)
• Keywords: Assistive communication, artificial intelligence, large language models, accessibility, human-computer interaction, language generation, real-time communication, customization, user research, personal expression

Research Background and Issues

• What problems or challenges did the authors identify?

  • AAC device users often require significant time to construct messages, leading to low communication efficiency.
  • Prolonged message construction increases physical and cognitive burdens.
  • Even with predictive technologies, users may struggle to respond quickly, making it difficult to participate in conversations.
  • Existing AAC devices lack support for personalized expression and offer limited interactive experiences.

• Why is this issue important?

  • Communication barriers for AAC users affect not only daily life but also social interactions and participation in group activities.
  • Improving AAC technologies can reduce users' physical and cognitive burdens, enhance quality of life, and promote the development of an inclusive society.

• Research Motivation and Related Work

  • Current static language prediction models and the acquisition of contextual data (e.g., photos) still have significant room for improvement.
  • Advanced large neural language models (e.g., GPT-3, BERT) demonstrate the potential to generate human-like text, which could help AAC users reduce input while producing more expressive outputs.
  • Preliminary studies suggest that LLMs can significantly reduce keystrokes, but improvements in customization and privacy control are needed for practical user adoption.

Solutions

• What methods or solutions did the authors propose?

  • The authors designed "Speech Macros" based on LLMs as a shortcut feature, allowing AAC users to generate complete sentences with minimal input.
  • A prototype system was developed to test its applicability in specific conversational scenarios.
  • Using a human-centered design research approach, the study explored how Speech Macros could reduce user input while maintaining communication style and contextual relevance.

• What is innovative about this solution?

  • The introduction of the "Speech Macros" concept enables users to customize shortcut inputs for different tasks, addressing specific communication challenges (e.g., expanding short messages, requesting help, responding to contextual questions).
  • Flexible application of LLMs' generative language capabilities, combined with a few examples and prompts, allows for quick adaptation to specific scenarios.
  • User research identified the need to control the style and contextual alignment of generated sentences while minimizing the interference of "over-automation" on users' personal expression.

• What are the implementation steps and key technologies used?

  1. Designed three types of Speech Macros (expanded replies, contextual replies, converting words into requests).
  2. Utilized the LaMDA language model with few-shot and zero-shot prompting for real-time generation, limiting the number of output sentences to optimize interface usability.
  3. Recruited 12 AAC device users to test the prototype in specific scenarios and provide feedback.
  4. Analyzed user interaction behaviors, feedback, and the quality of model-generated sentences to identify user needs and opportunities for improvement.

Research Outcomes

• What specific outcomes were achieved?

  • The study demonstrated that LLM-generated suggestions effectively reduced the physical and cognitive burdens of AAC users, enabling more efficient participation in interactions.
  • Appropriate Speech Macro functions (e.g., converting words into requests) were deemed highly practical by users.
  • Users expressed a desire for more personalized outputs and suggested participating in designing prompts to customize the system.

• What advantages does it have compared to existing solutions?

  • Speech Macros offer greater flexibility and adaptability to scenarios compared to traditional AAC prediction models.
  • The generative language capabilities of LLMs provide a wider variety of sentence suggestions with less input.
  • Users can explore and modify generated sentences to better express their personal style and contextual needs.

• What were the experimental or evaluation results?

  • Participants gave positive feedback on the three main macro functions (expanded replies, contextual replies, converting words into requests) to varying degrees.
  • During the experiment, users evaluated the social and personal relevance of the model-generated sentences, expressing a need for higher-quality social interactions.
  • Users raised concerns that erroneous suggestions could harm their social relationships or mislead communication partners.

• Limitations and Future Directions

  • Limitations:
    • The experiment duration was short, and user input samples were limited, preventing a comprehensive evaluation of model performance.
    • Existing LLMs were not specifically tailored to individuals, and output quality still requires improvement.
    • In high-pressure, high-risk scenarios (e.g., medical communication), the quality and accuracy of current suggestions may be insufficiently reliable.
  • Future Directions:
    • Design longer-term user studies to test the system's effects during sustained use.
    • Introduce more robust user configuration features, allowing users to easily adjust tone, length, or content preferences.
    • Develop privacy-friendly Speech Macro technologies to support offline processing of critical medical data.
    • Explore ways to integrate users' stored information to further reduce real-time input burdens while enhancing personalized design.