COMPA: Using Conversation Context to Achieve Common Ground in AAC

Title of the Paper

COMPA: Using Conversation Context to Achieve Common Ground in AAC

Paper Information

• Subject Area: Augmentative and Alternative Communication (AAC), Human-Computer Interaction, Accessibility Technology
• Keywords: AAC, Accessibility, Human-Computer Interaction, Conversation Modeling, Language Models, Group Communication, Interoperability, Assistive Tools

Research Background and Problem Statement

• Background: Augmentative and Alternative Communication (AAC) devices provide communication tools for individuals who cannot use speech effectively. However, due to the asymmetry between input and speech output speeds, AAC users often face challenges in dynamic group conversation environments.
• Existing Problems:
  • AAC users frequently struggle to join discussions in a timely manner while the conversation context is still relevant, leading to misunderstandings or missed opportunities to express themselves.
  • Current technologies, while attempting to improve communication efficiency through context prediction or phrase generation, fail to effectively support real-time context sharing in conversations.
  • Non-AAC users (conversation partners) often lack awareness of AAC users' intentions and communication needs, resulting in unequal exchanges.
• Research Motivation: To address the participation challenges faced by AAC users in group conversations, there is a need for technology that facilitates mutual understanding between AAC users and non-AAC users.
• Related Work:
  • Early developments include real-time communication support using visual cues and status indicators (e.g., typing status on devices).
  • Previous research has focused more on predictive models and semantic analysis, with less attention to solving the issue of shared understanding in conversations through context sharing.

Solution

• Core Approach: The study proposes and develops COMPA (Conversation Context for Mutual Participation in AAC), a browser extension that facilitates communication between AAC and non-AAC users through multi-layered support, including real-time conversation transcription and intent tagging.
• Innovations:
  • Introduces real-time conversation context marking, helping AAC users identify the specific parts of the conversation they wish to respond to.
  • Provides phrase initiation suggestions based on conversation context, enabling quick and contextually relevant responses.
  • Integrates intent grounding functionality, allowing AAC users to express their participation intentions (e.g., asking questions, replying, stating opinions).
• Implementation Steps:
  1. Develop a browser extension compatible with common video conferencing platforms like Google Meet.
  2. Implement multi-layered features: real-time text transcription, pause functionality, context marking, phrase suggestions, and intent notifications.
  3. Incorporate language models like ChatGPT to generate phrase suggestions directly reflecting the conversation content.
• Key Technologies:
  • Automatic speech transcription and context extraction.
  • Custom phrase initiation suggestions generated by language models.
  • User interface design tailored to the accessibility needs of AAC users.

Research Outcomes

• Specific Outcomes:
  • Developed and tested three versions of COMPA, each offering varying levels of context support (pause notifications, context marking, intent display).
  • Conducted group comparison experiments, demonstrating COMPA's effectiveness in enhancing AAC user participation and improving communication fluency.
  • Found that intent notifications and phrase suggestions directly improved non-AAC users' understanding of the conversation context.
• Comparative Advantages:
  • Unlike existing solutions that only serve AAC users, COMPA supports both AAC and non-AAC users.
  • Reduces the input burden on AAC users while enhancing conversation partners' awareness of context and intent.
• Experimental and Evaluation Results:
  • Among the 10 AAC users and their partners participating in the experiment, COMPA received high overall ratings: 4/5 participants expressed willingness to use certain versions in future online interactions.
  • The "pause transcription" and "context marking" features of COMPA were particularly well-received.
  • Although the intent buttons and phrase suggestions had a learning curve, some participants viewed them as potentially key features for improving conversations.
• Limitations and Future Directions:
  • The current study is limited in sample size and usage scenarios, focusing primarily on small group conversations and online meetings.
  • Long-term use and larger sample studies are needed to reveal the tool's potential impact more comprehensively.
  • Future work should enhance the system's adaptability to different user preferences and conversation scenarios, such as supporting more languages and multimodal input functionalities.

Conclusion and Future Outlook

As an innovative AAC assistive tool, COMPA has demonstrated a certain level of effectiveness in this study, providing a solution for mutual support between AAC and non-AAC users. With the advancement of multimodal and personalized technologies, future developments may further improve its user experience and applicability across diverse scenarios.