ImaginateAR: AI-Assisted In-Situ Authoring in Augmented RealityWhile augmented reality (AR) enables new ways to play, tell stories, and explore ideas rooted in the physical world, authoring personalized AR content remains difficult for non-experts, often requiring professional tools and time. Prior systems have explored AI-driven XR design but typically rely on manually defined VR environments and fixed asset libraries, limiting creative flexibility and real-world relevance. We introduce ImaginateAR, the first mobile tool for AI-assisted AR authoring to combine offline scene understanding, fast 3D asset generation, and LLMs—enabling users to create outdoor scenes through natural language interaction. For example, saying “a dragon enjoying a campfire” (P7) prompts the system to generate and arrange relevant assets, which can then be refined manually. Our technical evaluation shows that our custom pipelines produce more accurate outdoor scene graphs and generate 3D meshes faster than prior methods. A three-part user study (N=20) revealed preferred roles for AI, how users create in free-form use, and design implications for future AR authoring tools. ImaginateAR takes a step toward empowering anyone to create AR experiences anywhere—simply by speaking their imagination.2025JLJaewook Lee et al.AR Navigation & Context AwarenessGenerative AI (Text, Image, Music, Video)Interactive Narrative & Immersive StorytellingUIST
FlyMeThrough: Human-AI Collaborative 3D Indoor Mapping with Commodity DronesIndoor mapping data is crucial for routing, navigation, and building management, yet such data are widely lacking due to the manual labor and expense of data collection, especially for larger indoor spaces. Leveraging recent advancements in commodity drones and photogrammetry, we introduce FlyMeThrough---a drone-based indoor scanning system that efficiently produces 3D reconstructions of indoor spaces with human-AI collaborative annotations for key indoor points-of-interest (POI) such as entrances, restrooms, stairs, and elevators. We evaluated FlyMeThrough in 12 indoor spaces with varying sizes and functionality. To investigate use cases and solicit feedback from target stakeholders, we also conducted a qualitative user study with five building managers and five occupants. Our findings indicate that FlyMeThrough can efficiently and precisely create indoor 3D maps for strategic space planning, resource management, and navigation.2025XSXia Su et al.Geospatial & Map VisualizationDrone Interaction & ControlUIST
Accessibility Scout: Personalized Accessibility Scans of Built EnvironmentsAssessing the accessibility of unfamiliar built environments is critical for people with disabilities. However, manual assessments, performed by users or their personal health professionals, are laborious and unscalable, while automatic machine learning methods often neglect an individual user's unique needs. Recent advances in Large Language Models (LLMs) enable novel approaches to this problem, balancing personalization with scalability to enable more adaptive and context-aware assessments of accessibility. We present Accessibility Scout, an LLM-based accessibility scanning system that identifies accessibility concerns from photos of built environments. With use, Accessibility Scout becomes an increasingly capable "accessibility scout", tailoring accessibility scans to an individual's mobility level, preferences, and specific environmental interests through collaborative Human-AI assessments. We present findings from three studies: a formative study with six participants to inform the design of Accessibility Scout, a technical evaluation of 500 images of built environments, and a user study with 10 participants of varying mobility. Results from our technical evaluation and user study show that Accessibility Scout can generate personalized accessibility scans that extend beyond traditional ADA considerations. Finally, we conclude with a discussion on the implications of our work and future steps for building more scalable and personalized accessibility assessments of the physical world.2025WHWilliam Huang et al.Visual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Universal & Inclusive DesignPrivacy Perception & Decision-MakingUIST
ArtInsight: Enabling AI-Powered Artwork Engagement for Mixed Visual-Ability FamiliesWe introduce ArtInsight, a novel AI-powered system to facilitate deeper engagement with child-created artwork in mixed visual-ability families. ArtInsight leverages large language models (LLMs) to craft a respectful and thorough initial description of a child's artwork, and provides: creative AI-generated descriptions for a vivid overview, audio recording to capture the child's own description of their artwork, and a set of AI-generated questions to facilitate discussion between blind or low-vision (BLV) family members and their children. Alongside ArtInsight, we also contribute a new rubric to score AI-generated descriptions of child-created artwork and an assessment of state-of-the-art LLMs. We evaluated ArtInsight with five groups of BLV family members and their children, and as a case study with one BLV child therapist. Our findings highlight a preference for ArtInsight's longer, artistically-tailored descriptions over those generated by existing BLV AI tools. Participants highlighted the creative description and audio recording components as most beneficial, with the former helping "bring a picture to life" and the latter centering the child's narrative to generate context-aware AI responses. Our findings reveal different ways that AI can be used to support art engagement, including before, during, and after interaction with the child artist, as well as expectations that BLV adults and their sighted children have about AI-powered tools.2025ACArnavi Chheda-Kothary et al.Generative AI (Text, Image, Music, Video)AI-Assisted Creative WritingEmpowerment of Marginalized GroupsIUI
SPECTRA: Personalizable Sound Recognition for Deaf and Hard of Hearing Users through Interactive Machine LearningWe introduce SPECTRA, a novel pipeline for personalizable sound recognition designed to understand DHH users' needs when collecting audio data, creating a training dataset, and reasoning about the quality of a model. To evaluate the prototype, we recruited 12 DHH participants who trained personalized models for their homes. We investigated waveforms, spectrograms, interactive clustering, and data annotating to support DHH users throughout this workflow, and we explored the impact of a hands-on training session on their experience and attitudes toward sound recognition tools. Our findings reveal the potential for clustering visualizations and waveforms to enrich users' understanding of audio data and refinement of training datasets, along with data annotations to promote varied data collection. We provide insights into DHH users' experiences and perspectives on personalizing a sound recognition pipeline. Finally, we share design considerations for future interactive systems to support this population.2025SGSteven Goodman et al.University of Washington, Human Centered Design and EngineeringElectrical Muscle Stimulation (EMS)Visual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Deaf & Hard-of-Hearing Support (Captions, Sign Language, Vibration)CHI
Accessibility for Whom? Perceptions of Mobility Barriers Across Disability Groups and Implications for Designing Personalized MapsToday’s mapping tools fail to address the varied experiences of different mobility device users. This paper presents a large-scale online survey exploring how five mobility groups—users of canes, walkers, mobility scooters, manual wheelchairs, and motorized wheelchairs—perceive sidewalk barriers and differences therein. Using 52 sidewalk barrier images, respondents evaluated their confidence in navigating each scenario. Our findings (N=190) reveal variations in barrier perceptions across groups, while also identifying shared concerns. To further demonstrate the value of this data, we showcase its use in two custom prototypes: a visual analytics tool and a personalized routing tool. Our survey findings and open dataset advance work in accessibility-focused maps, routing algorithms, and urban planning.2025CLChu Li et al.University of Washington, Allen School of Computer ScienceUniversal & Inclusive DesignGeospatial & Map VisualizationPedestrian & Cyclist SafetyCHI
MobiPrint: A Mobile 3D Printer for Environment-Scale Design and Fabrication3D printing is transforming how we customize and create physical objects in engineering, accessibility, and art. However, this technology is still primarily limited to confined working areas and dedicated print beds thereby detaching design and fabrication from real-world environments and making measuring and scaling objects tedious and labor-intensive. In this paper, we present MobiPrint, a prototype mobile fabrication system that combines elements from robotics, architecture, and Human-Computer Interaction (HCI) to enable environment-scale design and fabrication in ad-hoc indoor environments. MobiPrint provides a multi-stage fabrication pipeline: first, the robotic 3D printer automatically scans and maps an indoor space; second, a custom design tool converts the map into an interactive CAD canvas for editing and placing models in the physical world; finally, the MobiPrint robot prints the object directly on the ground at the defined location. Through a "proof-by-demonstration" validation, we highlight our system's potential across different applications, including accessibility, home furnishing, floor signage, and art. We also conduct a technical evaluation to assess MobiPrint’s localization accuracy, ground surface adhesion, payload capacity, and mapping speed. We close with a discussion of open challenges and opportunities for the future of contextualized mobile fabrication.2024DZDaniel Campos Zamora et al.Deaf & Hard-of-Hearing Support (Captions, Sign Language, Vibration)Desktop 3D Printing & Personal FabricationShape-Changing Materials & 4D PrintingUIST
CookAR: Affordance Augmentations in Wearable AR to Support Kitchen Tool Interactions for People with Low VisionCooking is a central activity of daily living, supporting independence as well as mental and physical health. However, prior work has highlighted key barriers for people with low vision (LV) to cook, particularly around safely interacting with tools, such as sharp knives or hot pans. Drawing on recent advancements in computer vision (CV), we present CookAR, a head-mounted AR system with real-time object affordance augmentations to support safe and efficient interactions with kitchen tools. To design and implement CookAR, we collected and annotated the first egocentric dataset of kitchen tool affordances, fine-tuned an affordance segmentation model, and developed an AR system with a stereo camera to generate visual augmentations. To validate CookAR, we conducted a technical evaluation of our fine-tuned model as well as a qualitative lab study with 10 LV participants for suitable augmentation design. Our technical evaluation demonstrates that our model outperforms the baseline on our tool affordance dataset, while our user study indicates a preference for affordance augmentations over the traditional whole object augmentations.2024JLJaewook Lee et al.AR Navigation & Context AwarenessVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Deaf & Hard-of-Hearing Support (Captions, Sign Language, Vibration)UIST
SonifyAR: Context-Aware Sound Generation in Augmented RealitySound plays a crucial role in enhancing user experience and immersiveness in Augmented Reality (AR). However, current platforms lack support for AR sound authoring due to limited interaction types, challenges in collecting and specifying context information, and difficulty in acquiring matching sound assets. We present SonifyAR, an LLM-based AR sound authoring system that generates context-aware sound effects for AR experiences. SonifyAR expands the current design space of AR sound and implements a Programming by Demonstration (PbD) pipeline to automatically collect contextual information of AR events, including virtual-content-semantics and real-world context. This context information is then processed by a large language model to acquire sound effects with Recommendation, Retrieval, Generation, and Transfer methods. To evaluate the usability and performance of our system, we conducted a user study with eight participants and created five example applications, including an AR-based science experiment, and an assistive application for low-vision AR users.2024XSXia Su et al.AR Navigation & Context AwarenessGenerative AI (Text, Image, Music, Video)Context-Aware ComputingUIST
LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing SystemsCrowdsourcing platforms have transformed distributed problem-solving, yet quality control remains a persistent challenge. Traditional quality control measures, such as prescreening workers and refining instructions, often focus solely on optimizing economic output. This paper explores just-in-time AI interventions to enhance both labeling quality and domain-specific knowledge among crowdworkers. We introduce LabelAId, an advanced inference model combining Programmatic Weak Supervision (PWS) with FT-Transformers to infer label correctness based on user behavior and domain knowledge. Our technical evaluation shows that our LabelAId pipeline consistently outperforms state-of-the-art ML baselines, improving mistake inference accuracy by 36.7% with 50 downstream samples. We then implemented LabelAId into Project Sidewalk, an open-source crowdsourcing platform for urban accessibility. A between-subjects study with 34 participants demonstrates that LabelAId significantly enhances label precision without compromising efficiency while also increasing labeler confidence. We discuss LabelAId's success factors, limitations, and its generalizability to other crowdsourced science domains.2024CLChu Li et al.University of WashingtonExplainable AI (XAI)Crowdsourcing Task Design & Quality ControlCHI
GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented RealityVoice assistants (VAs) like Siri and Alexa are transforming human-computer interaction; however, they lack awareness of users' spatiotemporal context, resulting in limited performance and unnatural dialogue. We introduce GazePointAR, a fully-functional context-aware VA for wearable augmented reality that leverages eye gaze, pointing gestures, and conversation history to disambiguate speech queries. With GazePointAR, users can ask "what's over there?" or "how do I solve this math problem?" simply by looking and/or pointing. We evaluated GazePointAR in a three-part lab study (N=12): (1) comparing GazePointAR to two commercial systems, (2) examining GazePointAR's pronoun disambiguation across three tasks; (3) and an open-ended phase where participants could suggest and try their own context-sensitive queries. Participants appreciated the naturalness and human-like nature of pronoun-driven queries, although sometimes pronoun use was counter-intuitive. We then iterated on GazePointAR and conducted a first-person diary study examining how GazePointAR performs in-the-wild. We conclude by enumerating limitations and design considerations for future context-aware VAs.2024JLJaewook Lee et al.University of WashingtonEye Tracking & Gaze InteractionVoice User Interface (VUI) DesignAR Navigation & Context AwarenessCHI
Kinergy: Creating 3D Printable Motion using Embedded Kinetic EnergyWe present Kinergy—an interactive design tool for creating self-propelled motion by harnessing the energy stored in 3D printable springs. To produce controllable output motions, we introduce 3D printable kinetic units, a set of parameterizable designs that encapsulate 3D printable springs, compliant locks, and transmission mechanisms for three non-periodic motions—instant translation, instant rotation, continuous translation—and four periodic motions—continuous rotation, reciprocation, oscillation, intermittent rotation. Kinergy allows the user to create motion-enabled 3D models by embedding kinetic units, customize output motion characteristics by parameterizing embedded springs and kinematic elements, control energy by operating the specialized lock, and preview the resulting motion in an interactive environment. We demonstrate the potential of our techniques via example applications from spring-loaded cars to kinetic sculptures and close with a discussion of key challenges such as geometric constraints.2022LHLiang He et al.Shape-Changing Interfaces & Soft Robotic MaterialsDesktop 3D Printing & Personal FabricationUIST
ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing UsersRecent advances have enabled automatic sound recognition systems for deaf and hard of hearing (DHH) users on mobile devices. However, these tools use pre-trained, generic sound recognition models, which do not meet the diverse needs of DHH users. We introduce ProtoSound, an interactive system for customizing sound recognition models by recording a few examples, thereby enabling personalized and fine-grained categories. ProtoSound is motivated by prior work examining sound awareness needs of DHH people and by a survey we conducted with 472 DHH participants. To evaluate ProtoSound, we characterized performance on two real-world sound datasets, showing significant improvement over state-of-the-art (e.g., +9.7% accuracy on the first dataset). We then deployed ProtoSound's end-user training and real-time recognition through a mobile application and recruited 19 hearing participants who listened to the real-world sounds and rated the accuracy across 56 locations (e.g., homes, restaurants, parks). Results show that ProtoSound personalized the model on-device in real-time and accurately learned sounds across diverse acoustic contexts. We close by discussing open challenges in personalizable sound recognition, including the need for better recording interfaces and algorithmic improvements.2022DJDhruv Jain et al.University of Washington, GoogleDeaf & Hard-of-Hearing Support (Captions, Sign Language, Vibration)Motor Impairment Assistive Input TechnologiesCHI
Social, Environmental, and Technical: Factors at play in the current use and future design of small-group captioning Real-time captioning is a critical accessibility tool for many d/Deaf and hard of hearing (DHH) people. While the vast majority of captioning work has focused on formal settings and technical innovations, in contrast, we investigate captioning for informal, interactive small-group conversations, which have a high degree of spontaneity and foster dynamic social interactions. This paper reports on semi-structured interviews and design probe activities we conducted with 15 DHH participants to understand their use of existing real-time captioning services and future design preferences for both in-person and remote small-group communication. We found that our participants’ experiences of captioned small-group conversations are shaped by social, environmental, and technical considerations (e.g., interlocutors’ pre-established relationships, the type of captioning displays available, and how far captions lag behind speech). When considering future captioning tools, participants were interested in greater feedback on non-speech elements of conversation (e.g., speaker identity, speech rate, volume) both for their personal use and to guide hearing interlocutors towards more accessible communication. We contribute a qualitative account of DHH people’s real-time captioning experiences during small-group conversation and future design considerations to better support the groups being captioned, both in person and online.2021EMEmma J McDonnell et al.Accessibility and Assistive TechnologiesCSCW
Examining Visual Semantic Understanding in Blind and Low-Vision Technology UsersVisual semantics provide spatial information like size, shape, and position, which are necessary to understand and efficiently use interfaces and documents. Yet little is known about whether blind and low-vision (BLV) technology users want to interact with visual affordances, and, if so, for which task scenarios. In this work, through semi-structured and task-based interviews, we explore preferences, interest levels, and use of visual semantics among BLV technology users across two device platforms (smartphones and laptops), and information seeking and interactions common in apps and web browsing. Findings show that participants could benefit from access to visual semantics for collaboration, navigation, and design. To learn this information, our participants used trial and error, sighted assistance, and features in existing screen reading technology like touch exploration. Finally, we found that missing information and inconsistent screen reader representations of user interfaces hinder learning. We discuss potential applications and future work to equip BLV users with necessary information to engage with visual semantics.2021VPVenkatesh Potluri et al.University of WashingtonVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)CHI
What Do We Mean by "Accessibility Research"? A Systematic Review of Accessibility Papers in CHI and ASSETS from 1994 to 2019Accessibility research has grown substantially in the past few decades, yet there has been no literature review of the field. To understand current and historical trends, we created and analyzed a dataset of accessibility papers appearing at CHI and ASSETS since ASSETS' founding in 1994. We qualitatively coded areas of focus and methodological decisions for the past 10 years (2010-2019, N=506 papers), and analyzed paper counts and keywords over the full 26 years (N=836 papers). Our findings highlight areas that have received disproportionate attention and those that are underserved--for example, over 43% of papers in the past 10 years are on accessibility for blind and low vision people. We also capture common study characteristics, such as the roles of disabled and nondisabled participants as well as sample sizes (e.g., a median of 13 for participant groups with disabilities and older adults). We close by critically reflecting on gaps in the literature and offering guidance for future work in the field.2021KMKelly Mack et al.University of WashingtonVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Deaf & Hard-of-Hearing Support (Captions, Sign Language, Vibration)Cognitive Impairment & Neurodiversity (Autism, ADHD, Dyslexia)CHI
Urban Accessibility as a Socio-Political Problem: A Multi-Stakeholder AnalysisTraditionally, urban accessibility is defined as the ease of reaching destinations. Studies on urban accessibility for pedestrians with mobility disabilities (e.g., wheelchair users) have primarily focused on understanding the challenges that the built environment imposes and how they overcome them. In this paper, we move beyond physical barriers and focus on socio-political challenges in the civic ecosystem that impedes accessible infrastructure development. Using a multi-stakeholder approach, we interviewed five primary stakeholder groups (N=25): (1) people with mobility disabilities, (2) caregivers, (3) accessibility advocates, (4) department officials, and (5) policymakers. We discussed their current accessibility assessment and decision-making practices. We identified the key needs and desires of each group, how they differed, and how they interacted with each other in the civic ecosystem to bring about change. We found that people, politics, and money were intrinsically tied to underfunded accessibility improvement projects—without continued support from the public and the political leadership, existing funding may also disappear. Using the insights from these interviews, we explore how may technology enhance our stakeholders’ decision-making processes and facilitate accessible infrastructure development.2020MSManaswi Saha et al.Accessibility / Women's EmpowermentCSCW
ARMath: Augmenting Everyday Life with Math LearningWe introduce ARMath, a mobile Augmented Reality (AR) system that allows ch ildren to discover mathematical concepts in familiar, ord inary objects and engage with math problems in meaningful contexts. Leveraging advanced computer vision, ARMath recognizes everyday objects, visualizes their mathematical attributes, and turns them into tangible or virtual manipulatives. Using the manipulatives, children can solve problems that situate math operations or concepts in specific everyday contexts. Informed by four participatory design sessions with teachers and children, we developed five ARMath modules to support basic arithmetic and 2D geometry. We also conducted an exploratory evaluation of ARMath with 27 children (ages 5-8) at a local children's museum. Our findings demonstrate how ARMath engages children in math learning, how failures in AI can be used as learning opportunities, and challenges that children face when using ARMath.2020SKSeokbin kang et al.University of MarylandAR Navigation & Context AwarenessK-12 Digital Education ToolsSTEM Education & Science CommunicationCHI
HomeSound: An Iterative Field Deployment of an In-Home Sound Awareness System for Deaf or Hard of Hearing UsersWe introduce HomeSound, an in-home sound awareness system for Deaf and hard of hearing (DHH) users. Similar to the Echo Show or Nest Hub, HomeSound consists of a microphone and display, and uses multiple devices installed in each home. We iteratively developed two prototypes, both of which sense and visualize sound information in real-time. Prototype 1 provided a floorplan view of sound occurrences with waveform histories depicting loudness and pitch. A three-week deployment in four DHH homes showed an increase in participants' home- and self-awareness but also uncovered challenges due to lack of line of sight and sound classification. For Prototype 2, we added automatic sound classification and smartwatch support for wearable alerts. A second field deployment in four homes showed further increases in awareness but misclassifications and constant watch vibrations were not well received. We discuss findings related to awareness, privacy, and display placement and implications for future home sound awareness technology.2020DJDhruv Jain et al.University of WashingtonDeaf & Hard-of-Hearing Support (Captions, Sign Language, Vibration)Smart Home Privacy & SecurityCHI
Ondulé: Designing and Controlling 3D Printable SpringsWe present Ondulé, a novel computational design tool to add elastic deformation behaviors to static 3D models using a combination of 3D-printed springs and mechanical joints. Springs are unique because they can exert expressive deformation behaviors and store mechanical energy. Informed by spring theory and our empirical mechanical experiments, we introduce spring and joint-based design techniques that support a range of parameterizable deformation behaviors, including compress, extend, twist, bend, and various combinations. To enable users to design and add these 3D-printable deformations to their models, we introduce a custom spring design tool for Rhino. Here, users can convert selected geometries into springs, customize spring stiffness, and parameterize their design to obtain a desired deformation behavior. To demonstrate the feasibility of our approach and the breadth of new 3D-printable designs that it enables, we showcase a set of example applications from launching rocket toys to tangible storytelling props. We conclude with a discussion of key challenges and open research questions.2019LHLiang He et al.Shape-Changing Interfaces & Soft Robotic MaterialsShape-Changing Materials & 4D PrintingUIST