SignGlass: First-Person View Comprehensive and Generalizable ASL Translation Using Wearable Glass

DOI

Communication barriers between Deaf and Hard-of-Hearing (DHH) individuals and hearing individuals remain a major challenge, highlighting the need for technologies that enable seamless sign language interpretation. However, current American Sign Language (ASL) recognition and translation systems face key limitations, including poor portability, complex usage settings, incomplete capture of essential components, and weak generalization, reducing practicality and user acceptance. To address these challenges, we present SignGlass, the first smart glasses equipped with three wearable cameras for comprehensive capture of both manual and non-manual ASL markers, supported by advanced algorithms for real-time recognition and translation into English. Specifically, SignGlass integrates a jitter-aware spatio-temporal attention mechanism for robust recognition of hand movement patterns. Complementing this, a dual-camera, temporally-aware facial module captures the subtle facial expressions essential for ASL comprehension. To support diverse signing styles across individuals, we further introduce a cascaded data augmentation strategy to improve model generalization. In a user study with 14 Deaf participants, SignGlass achieved high translation accuracy and was well-received, demonstrating its effectiveness in bridging communication gaps. This work highlights the promise of multi-camera wearable systems in advancing ASL translation and promoting more accessible communication.