Beyond Accuracy: Auditing Allocative Harms in Facial-Gesture Recognition for People with Motor Impairments

Paper Title

Beyond Accuracy: Auditing Allocative Harms in Facial-Gesture Recognition for People with Motor Impairments

Publication Info

• Topic area: Fairness and inclusivity in facial-gesture recognition systems for motor-impaired users.
• Keywords: Facial gesture recognition, motor impairments, allocative harm, sensorimotor alignment, algorithmic fairness, perception gap, motion trajectories, inclusive design, assistive technology, accessibility.

Background and Problem

• Problem / challenge: Current facial-gesture recognition models are trained predominantly on able-bodied datasets, assuming normative motor control and proprioception. This results in systematic misclassification or exclusion of gestures performed by people with motor impairments (PwM), leading to allocative harms.
• Significance: Facial-gesture interfaces are critical for hands-free interaction, particularly for PwM. Ensuring inclusivity and robustness in these systems is essential for equitable access to technology.
• Motivation and related work: Prior research has focused on demographic fairness (e.g., race, gender) in facial analysis but has largely ignored sensorimotor diversity. Existing models fail to account for atypical, asymmetric, or low-amplitude gestures common in PwM, creating a gap in inclusive design and evaluation.

Solution

• Proposed approach: FairGesture, a diagnostic auditing method designed to quantify and interpret intention–recognition mismatches in facial-gesture recognition systems.
• Novelty:
  1. Introduction of the Perception Gap (PG) metric to measure discrepancies between user-perceived success and model recognition.
  2. Motion-dimension analysis to identify execution deviations (e.g., amplitude, direction, timing, activation region).
  3. Integration of user perception accounts to contextualize mismatches and inform inclusive design.
  4. Release of actionable auditing assets, including scripts, visualizations, and interpretive tools.
• Procedure and key techniques:
  • Conducted a mixed-methods study with 22 participants (11 PwM, 11 able-bodied controls) performing 37 facial gestures.
  • Collected recognition outputs from MediaPipe and OpenFace, self-evaluation scores, motion trajectories, and qualitative feedback.
  • Computed PG scores and analyzed motion trajectories across four dimensions: amplitude, direction, temporal dynamics, and activation region.
  • Triangulated quantitative and qualitative data to identify patterns of intention–recognition mismatch.

Results

• Concrete findings:
  • Recognition accuracy for PwM was significantly lower (MediaPipe: 56.2% ± 26.8%; OpenFace: 53.94%) compared to able-bodied participants (MediaPipe: 74.8% ± 16.5%; OpenFace: 77.03%).
  • Positive Perception Gaps (PGs) were concentrated in gestures involving low amplitude, asymmetry, and directional instability, indicating systematic intention–recognition mismatches.
  • PwM self-reported high confidence in gesture execution (mean self-evaluation score: 7.12 ± 1.74), despite frequent misrecognitions.
• Advantage over baselines:
  • Cross-model consistency in mismatch patterns (MediaPipe and OpenFace) suggests that issues are not model-specific but systemic.
  • FairGesture provides actionable insights for addressing allocative harms, unlike conventional accuracy-focused evaluations.
• Experiments / evaluation:
  • Mixed-methods study with 22 participants (11 PwM, 11 able-bodied controls).
  • Metrics: recognition accuracy, self-evaluation scores, Perception Gap, motion trajectories.
  • Statistical tests: Two-sample t-tests, Mann-Whitney U tests, and Benjamini-Hochberg FDR correction.
• Limitations and future work:
  • Limited participant diversity (e.g., exclusion of older adults and individuals with cognitive impairments).
  • Focus on one-bit gestures; more complex gestures were excluded.
  • Moderate inter-rater agreement in qualitative annotations (Cohen’s κ = 0.69).
  • Future work will extend FairGesture to vision-language models (VLMs) and other user groups.

Summary

This paper introduces FairGesture, a diagnostic method to audit allocative harms in facial-gesture recognition systems for people with motor impairments. Through a mixed-methods study, the authors reveal systematic intention–recognition mismatches caused by sensorimotor diversity, with recognition accuracy significantly lower for PwM. The Perception Gap metric, motion-dimension analysis, and user perception accounts provide actionable insights for inclusive model and interface design. The findings highlight the need to reframe gesture recognition as a sensorimotor alignment problem, advocating for systems that accommodate motor diversity rather than treating it as noise.