Co-Designing QuickPic: Automated Topic-Specific Communication Boards from Photographs for AAC-Based Language Instruction

Title of the Paper

Co-Designing QuickPic: Automated Topic-Specific Communication Boards from Photographs for AAC-Based Language Instruction

Paper Information

• Research Area: Augmentative and Alternative Communication (AAC), Artificial Intelligence, Special Education, Speech Therapy
• Keywords: Augmentative and Alternative Communication, Autism, Just-in-Time Support, Large Language Models (LLM), Assistive Technology

Research Background and Problem

• Problem or Challenge:

  • Current topic-specific AAC communication boards require manual vocabulary programming, which is time-consuming and labor-intensive, making it difficult for speech therapists and special education teachers to widely adopt them in practice.
  • Existing AI-based technologies have demonstrated potential for automatically generating communication boards but have not fully explored the practical needs and design improvements required for AAC language instruction.
  • Key obstacles include the lack of solutions for dynamic generation and immediate language support. Existing methods (e.g., Click AAC) fail to adequately meet professionals' requirements for user interface and vocabulary generation quality.

• Importance:

  • AAC communication boards are essential for non-verbal users (e.g., children with autism) to learn symbolic language and develop expressive abilities.
  • Automation can reduce workload, increase interaction time for speech therapists and educators, and improve family acceptance of AAC devices.

• Motivation and Related Work:

  • Investigating strategies to improve existing AI-based methods, including optimizing vocabulary generation using more advanced large language models (e.g., GPT).
  • Designing specialized AAC tools to support professionals in creating dynamic language content tailored to specific educational scenarios.

Solution

• Proposed Method or Solution:

  • Develop QuickPic, a mobile AAC application capable of automatically generating topic-specific communication boards from photographs, with users able to quickly edit the generated content.
  • Optimize vocabulary generation quality using GPT-3.5 and compare it with the VIST method used in Click AAC.
  • Employ a co-design approach to integrate the professional needs of speech therapists and special education teachers into the interface and core functionalities.

• Innovations:

  • Provide "just-in-time" language support functionality to significantly simplify the process of generating and editing communication boards.
  • Integrate automated management and customization of symbol libraries (e.g., Picture Communication Symbols, PCS).
  • Utilize GPT models to generate more relevant vocabulary, addressing limitations of existing technologies in supporting educational and therapeutic scenarios effectively.

• Implementation Steps and Key Technologies:

  • Design Process:
    1. Identify the primary functional requirements of the new tool through research and design methods.
    2. Develop a low-fidelity prototype featuring key interfaces and interaction logic.
    3. Develop and test a high-fidelity prototype (QuickPic v.0.1).
    4. Conduct field testing and gather user feedback to iterate and optimize the interface and generation methods.
  • Technological Applications:
    • Enhance image analysis models to produce high-quality photo descriptions.
    • Use ChatGPT to optimize vocabulary generation, ensuring the generated symbols are highly relevant to the context.

Research Outcomes

• Specific Results:

  • Developed the QuickPic application, integrating automated vocabulary generation and user-friendly editing features.
  • Implemented a GPT-based vocabulary generation method, significantly improving vocabulary relevance and quality compared to the VIST method.
  • Surveyed speech therapists and special education teachers on their satisfaction with QuickPic, demonstrating its efficiency in classroom and therapeutic settings.

• Advantages Compared to Existing Solutions:

  • QuickPic significantly reduces the time required for professionals to create communication boards.
  • Vocabulary generated by the GPT method is more closely aligned with user-specific topics, greatly expanding language learning possibilities.
  • The flexibility and intuitiveness of the interface received high praise from users.

• Experimental or Evaluation Results:

  • The GPT method outperformed VIST in generating relevant vocabulary, particularly for objects, adjectives, and prepositions.
  • 95% of participants found QuickPic easier to operate and edit compared to existing AAC tools (e.g., Boardmaker or TouchChat HD-AAC).
  • QuickPic scored significantly higher on the MAUQ questionnaire than traditional tools, indicating excellent usability.

• Limitations and Future Directions:

  • Limitations:
    • Vocabulary generation may still be constrained by scripted language models (e.g., lack of diversity or adaptability to certain scenarios).
    • Automated symbol association may require further optimization (e.g., distinguishing word types).
  • Future Directions:
    • Optimize generation models to better adapt to user scenarios, such as fine-tuning models based on image content.
    • Study the long-term impact of QuickPic on AAC users and families, including communication skills and learning outcomes.
    • Explore ethical considerations and balance the applicability of AI-generated vocabulary in family and educational environments.

Conclusion

This paper introduces an innovative AAC tool, QuickPic, which leverages co-design and technological optimization to address the dynamic vocabulary needs of language therapy and special education scenarios. The research demonstrates that by integrating large language models, the tool surpasses existing solutions in vocabulary generation quality and user experience, providing significant application insights and future research opportunities for the AAC field.