Grab-and-Release Spelling in XR: A Feasibility Study for Nonspeaking Autistic People Using Video-Passthrough Devices

This paper explores the feasibility of using video-passthrough Extended Reality (XR) devices to support communication in nonspeaking autistic individuals. Prior XR work relied on expensive AR headsets and near-hand tapping interactions. We present LetterBox, a novel application for video-passthrough XR headsets (e.g., Meta Quest) that enables spelling via a “grab-snap-release” interaction. The app includes three immersion levels and a dynamic pass-through window to maintain caregiver presence. We conducted a study with 19 participants across four North American sites. All completed a multiphase spelling task and answered open-ended questions. Despite tolerability concerns, all participants wore the headset throughout; only one requested a break. The average spelling accuracy was 90.91%. In open-spelling, 14 participants responded—often independently. Reaction time and interaction speed data highlighted the impact of visual complexity, offering insights for reducing errors. These findings suggest video-passthrough XR is well tolerated and that grab-snap-release interactions may benefit users with motor challenges.