Disability-First AI Dataset Annotation: Co-designing Stuttered Speech Annotation Guidelines with People Who Stutter

Paper Title

Disability-First AI Dataset Annotation: Co-designing Stuttered Speech Annotation Guidelines with People Who Stutter

Publication Info

• Topic area: AI dataset annotation for accessibility, focusing on stuttered speech.
• Keywords: stuttered speech, accessibility datasets, disability-first, AI annotation, embodied knowledge, speech disfluencies, participatory design, stuttering annotation, human-computer interaction, inclusive AI.

Background and Problem

• Problem / challenge: Accessibility datasets often suffer from inconsistent or inaccurate annotations due to the lack of disability-specific expertise among annotators. For stuttered speech, this results in mislabeling and low inter-rater agreement, even among trained professionals.
• Significance: Accurate and representative annotations are critical for improving AI systems’ performance for disabled users, particularly in speech-to-text applications and other accessibility technologies.
• Motivation and related work: Existing datasets like Sep-28k and VizWiz highlight significant annotation inconsistencies due to annotators’ limited understanding of disability-specific needs. Prior efforts in disability-first datasets have focused on data collection, but annotation processes remain underexplored. This paper addresses this gap by involving people who stutter (PWS) in the annotation process.

Solution

• Proposed approach: A disability-first, co-designed annotation framework for stuttered speech, developed collaboratively with PWS and speech-language pathologists (SLPs).
• Novelty:
  1. Introduced the first PWS-centered stuttered speech annotation guidelines.
  2. Integrated embodied knowledge of PWS into the annotation process.
  3. Proposed a multiplicity-aware approach to account for the subjectivity and complexity of stuttering.
• Procedure and key techniques:
  1. Formative studies: Reviewed existing datasets (e.g., Sep-28k) and conducted interviews with PWS AI professionals to identify annotation challenges.
  2. Co-design sessions: Iteratively developed and refined guidelines with PWS AI professionals and SLPs, focusing on stuttering event definitions, annotation consistency, and subjective interpretation.
  3. Evaluation sessions: Reviewed annotated speech samples with PWS contributors to assess the framework’s applicability and gather feedback.

Results

• Concrete findings:
  • Annotation disagreements between Sep-28k and Sep-28k-SW showed 12.32% false positives for prolongations and 12.13% for blocks.
  • Participants emphasized the importance of non-verbal cues (e.g., breathing patterns) and subjective experiences in identifying stuttering events.
  • Updated guidelines included five stuttering event types (blocks, prolongations, sound repetitions, word/phrase repetitions, interjections) and strategies for handling mixed events.
• Advantage over baselines:
  • Improved annotation accuracy by centering PWS expertise.
  • Addressed inherent subjectivity in stuttering perception through reflexive practices and iterative refinements.
• Experiments / evaluation:
  • Conducted qualitative analysis of co-design and evaluation sessions with PWS and SLPs.
  • Participants reviewed annotated speech samples and provided feedback on label accuracy and applicability across scenarios (e.g., therapy, auto-captioning).
• Limitations and future work:
  • Small participant sample skewed toward higher socio-economic and technical literacy backgrounds.
  • Guidelines influenced by pre-existing frameworks; further refinement needed with broader community involvement.
  • Limited coverage of diverse English dialects and other languages.

Summary

This study introduces a disability-first approach to annotating stuttered speech datasets, emphasizing the embodied knowledge of PWS. Through co-design sessions with PWS and SLPs, the authors developed annotation guidelines that address the subjectivity and complexity of stuttering. The guidelines include five stuttering event types and strategies for handling mixed events, with a focus on non-verbal cues and context-dependent evaluations. While the approach improves annotation accuracy and inclusivity, future work is needed to expand participant diversity and adapt the framework for other languages and speech patterns. This work demonstrates the importance of centering disabled expertise throughout the AI development pipeline.