AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multimodal Information Between Reality and VideosVideos offer rich audiovisual information that can support people in performing activities of daily living (ADLs), but they remain largely inaccessible to blind or low-vision (BLV) individuals. In cooking, BLV people often rely on non-visual cues---such as touch, taste, and smell---to navigate their environment, making it difficult to follow the predominantly audiovisual instructions found in video recipes. To address this problem, we introduce AROMA, an AI system that provides timely responses to the user based on real-time, context-aware assistance by integrating non-visual cues perceived by the user, a wearable camera feed, and video recipe content. AROMA uses a mixed-initiative approach: it responds to user requests while also proactively monitoring the video stream to offer timely alerts and guidance. This collaborative design leverages the complementary strengths of the user and AI system to align the physical environment with the video recipe, helping the user interpret their current cooking state and make sense of the steps. We evaluated AROMA through a study with eight BLV participants and offered insights for designing interactive AI systems to support BLV individuals in performing ADLs.2025ZNZheng Ning et al.Conversational ChatbotsVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Context-Aware ComputingUIST
“It’s the only thing I can trust”: Envisioning Large Language Model Use by Autistic Workers for Communication AssistanceAutistic adults often experience stigma and discrimination at work, leading them to seek social communication support from coworkers, friends, and family despite emotional risks. Large language models (LLMs) are increasingly considered an alternative. In this work, we investigate the phenomenon of LLM use by autistic adults at work and explore opportunities and risks of LLMs as a source of social communication advice. We asked 11 autistic participants to present questions about their own workplace-related social difficulties to (1) a GPT-4-based chatbot and (2) a disguised human confederate. Our evaluation shows that participants strongly preferred LLM over confederate interactions. However, a coach specializing in supporting autistic job-seekers raised concerns that the LLM was dispensing questionable advice. We highlight how this divergence in participant and practitioner attitudes reflects existing schisms in HCI on the relative privileging of end-user wants versus normative good and propose design considerations for LLMs to center autistic experiences.2024JJJiWoong Jang et al.Carnegie Mellon UniversityHuman-LLM CollaborationCognitive Impairment & Neurodiversity (Autism, ADHD, Dyslexia)Empowerment of Marginalized GroupsCHI
A Contextual Inquiry of People with Vision Impairments in CookingIndividuals with vision impairments employ a variety of strategies for object identification, such as pans or soy sauce, in the culinary process. In addition, they often rely on contextual details about objects, such as location, orientation, and current status, to autonomously execute cooking activities. To understand how people with vision impairments collect and use the contextual information of objects while cooking, we conducted a contextual inquiry study with 12 participants in their own kitchens. This research aims to analyze object interaction dynamics in culinary practices to enhance assistive vision technologies for visually impaired cooks. We outline eight different types of contextual information and the strategies that blind cooks currently use to access the information while preparing meals. Further, we discuss preferences for communicating contextual information about kitchen objects as well as considerations for the deployment of AI-powered assistive technologies.2024FLFranklin Mingzhe Li et al.Carnegie Mellon UniversityVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Context-Aware ComputingCHI
Breaking the ``Inescapable'' Cycle of Pain: Supporting Wheelchair Users' Upper Extremity Health Awareness and Management with Tracking TechnologiesUpper extremity (UE) health issues are a common concern among wheelchair users and have a large impact on their independence, social participation, and quality of life. However, despite the well-documented prevalence and negative impacts, these issues remain unresolved. Existing solutions (e.g. surgical repair, conservative treatments) often fail to promote sustained UE health improvement in wheelchair users' day-to-day lives. Recent HCI research has shown the effectiveness of health tracking technologies in supporting patients' self-care for different health conditions (e.g. chronic diseases, mental health). In this work, we explore how health tracking technologies could support wheelchair users' UE health self-care. We conducted semi-structured interviews with 12 wheelchair users and 5 therapists to understand their practices and challenges in UE health management, as well as the potential benefits of integrating health tracking technologies into self-care routines. We discuss design implications for UE health tracking technologies and outline opportunities for future investigation.2023YLYunzhi Li et al.Carnegie Mellon UniversityMotor Impairment Assistive Input TechnologiesMental Health Apps & Online Support CommunitiesFitness Tracking & Physical Activity MonitoringCHI
Understanding Visual Arts Experiences of Blind PeopleVisual arts play an important role in cultural life and provide access to social heritage and self-enrichment, but most visual arts are inaccessible to blind people. Researchers have explored different ways to enhance blind people's access to visual arts (e.g., audio descriptions, tactile graphics). However, how blind people adopt these methods remains unknown. We conducted semi-structured interviews with 15 blind visual arts patrons to understand how they engage with visual artwork and the factors that influence their adoption of visual arts access methods. We further examined interview insights in a follow-up survey (N=220). We present: 1) current practices and challenges of accessing visual artwork in-person and online (e.g., Zoom tour), 2) motivation and cognition of perceiving visual arts (e.g., imagination), and 3) implications for designing visual arts access methods. Overall, our findings provide a roadmap for technology-based support for blind people's visual arts experiences.2023FLFranklin Mingzhe Li et al.Carnegie Mellon UniversityVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Museum & Cultural Heritage DigitizationCHI
An Exploration of Captioning Practices and Challenges of Individual Content Creators on YouTube for People with Hearing ImpairmentsDeaf and Hard-of-Hearing (DHH) audiences have long complained about caption qualities for many online videos created by individual content creators on video-sharing platforms (e.g., YouTube). However, there lack explorations of practices, challenges, and perceptions of online video captions from the perspectives of both individual content creators and DHH audiences. In this work, we first explore DHH audiences' feedback on and reactions to YouTube video captions through interviews with 13 DHH individuals, and uncover DHH audiences' experiences, challenges, and perceptions on watching videos created by individual content creators (e.g., manually added caption tags could create additional confidence and trust in caption qualities for DHH audiences). We then discover individual content creators' practices, challenges, and perceptions on captioning their videos (e.g., back-captioning problems) by conducting a YouTube video analysis with 189 captioning-related YouTube videos, followed by a survey with 62 individual content creators. Overall, our findings provide an in-depth understanding of captions generated by individual content creators and bridge the knowledge gap mutually between content creators and DHH audiences on captions.2022FLFranklin Mingzhe Li et al.Accessibility; AccessibilityCSCW
"It Feels Like Taking a Gamble": Exploring Perceptions, Practices, and Challenges of Using Makeup and Cosmetics for People with Visual ImpairmentsMakeup and cosmetics offer the potential for self-expression and the reshaping of social roles for visually impaired people. However, there exist barriers to conducting a beauty regime because of the reliance on visual information and color variances in makeup. We present a content analysis of 145 YouTube videos to demonstrate visually impaired individuals' unique practices before, during, and after doing makeup. Based on the makeup practices, we then conducted semi-structured interviews with 12 visually impaired people to discuss their perceptions of and challenges with the makeup process in more depth. Overall, through our findings and discussion, we present novel perceptions of makeup from visually impaired individuals (e.g., broader representations of blindness and beauty). The existing challenges provide opportunities for future research to address learning barriers, insufficient feedback, and physical and environmental barriers, making the experience of doing makeup more accessible to people with visual impairments.2022FLFranklin Mingzhe Li et al.Carnegie Mellon UniversityVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Universal & Inclusive DesignCHI
Accessibility and The Crowded Sidewalk: Micromobility’s Impact on Public SpaceOver the past several years, micromobility devices—small-scale, networked vehicles used to travel short distances—have begun to pervade cities, bringing promises of sustainable transportation and decreased congestion. Though proponents herald their role in offering lightweight solutions to disconnected transit, smart scooters and autonomous delivery robots increasingly occupy pedestrian pathways, reanimating tensions around the right to public space. Drawing on interviews with disabled activists, government officials, and commercial representatives, we chart how devices and policies co-evolve to fulfill municipal sustainability goals, while creating obstacles for people with disabilities whose activism has long resisted inaccessible infrastructure. We reflect on efforts to redistribute space, institute tech governance, and offer accountability to those who involuntarily encounter interventions on the ground. In studying micromobility within spatial and political context, we call for the HCI community to consider how innovation transforms as it moves out from centers of development toward peripheries of design consideration.2021CBCynthia L Bennett et al.Micromobility (E-bike, E-scooter) InteractionCommunity Engagement & Civic TechnologyDIS
From Tactile to NavTile: Opportunities and Challenges for Multi-Modal Feedback in Guiding Surfaces during Non-Visual NavigationTactile guiding surfaces in the built environment have held a contentious place in the process of navigation by people who are blind or visually impaired. Despite standards for tactile guiding surfaces, problems persist with inconsistent implementation, perception, and geographic orientation. We investigate the role of tactile cues in non-visual navigation and attitudes surrounding guiding surfaces through a survey of 67 people with vision impairments and ten interviews with navigation and public accessibility experts. Our participants revealed several opportunities to augment existing tactile surfaces while envisioning novel multimodal feedback solutions in immediately relevant contexts. We also propose an approach for designing and exploring low cost, multimodal tactile surfaces, which we call navtiles. Finally, we discuss practical aspects of implementation for new design alternatives such as standardization, installation, movability, discoverability, and a need for transparency. Collectively, these insights contribute to the production and implementation of novel multimodal navigation aids.2021SSSaiganesh Swaminathan et al.Carnegie Mellon UniversityVibrotactile Feedback & Skin StimulationDeaf & Hard-of-Hearing Support (Captions, Sign Language, Vibration)Context-Aware ComputingCHI
What Makes Videos Accessible to Blind and Visually Impaired People?Videos on sites like YouTube have become a primary source for information online. User-generated videos almost universally lack audio descriptions, making most videos inaccessible to blind and visually impaired (BVI) consumers. Our formative studies with BVI people revealed that they used a time-consuming trial-and-error approach when searching for videos: clicking on a video, watching a portion, leaving the video, and repeating the process to find videos that would be accessible — or understandable without additional description of the visual content. BVI people also reported video accessibility heuristics that characterize accessible and inaccessible videos. We instantiate 7 of the identified heuristics (2 audio-related, 2 video-related, and 3 audio-visual) as automated metrics to assess video accessibility. Our automated video accessibility metrics correlate with BVI people’s perception of video accessibility (Adjusted R-squared = 0.642). We augment a video search interface with our video accessibility metrics and find that our system improves BVI peoples’ efficiency in finding accessible videos. With accessibility metrics, participants found videos 40% faster and clicked 54% less videos in our user study. By integrating video accessibility metrics, video hosting platforms could help people surface accessible videos and encourage content creators to author more accessible products, improving video accessibility for all.2021XLXingyu Liu et al.Carnegie Mellon UniversityVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)CHI
Twitter A11y: A Browser Extension to Make Twitter Images AccessibleSocial media platforms are integral to public and private discourse, but are becoming less accessible to people with vision impairments due to an increase in user-posted images. Some platforms (i.e. Twitter) let users add image descriptions (alternative text), but only 0.1% of images include these. To address this accessibility barrier, we created Twitter A11y, a browser extension to add alternative text on Twitter using six methods. For example, screenshots of text are common, so we detect textual images, and create alternative text using optical character recognition. Twitter A11y also leverages services to automatically generate alternative text or reuse them from across the web. We compare the coverage and quality of Twitter A11y's six alt-text strategies by evaluating the timelines of 50 self-identified blind Twitter users. We find that Twitter A11y increases alt-text coverage from 7.6% to 78.5%, before crowdsourcing descriptions for the remaining images. We estimate that 57.5% of returned descriptions are high-quality. We then report on the experiences of 10 participants with visual impairments using the tool during a week-long deployment. Twitter A11y increases access to social media platforms for people with visual impairments by providing high-quality automatic descriptions for user-posted images.2020CGCole Gleason et al.Carnegie Mellon UniversityVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)CHI
"Occupational Therapy is Making": Clinical Rapid Prototyping and Digital FabricationConsumer-fabrication technologies potentially improve the effectiveness and adoption of assistive technology (AT) by engaging AT users in AT creation. However, little is known about the role of clinicians in this revolution. We investigate clinical AT fabrication by working as expert fabricators for clinicians over a four-month period. We observed and co-designed AT with four occupational therapists at two clinics: a free clinic for uninsured clients, and a Veteran's Affairs Hospital. We find that existing fabrication processes, particularly with respect to rapid prototyping, do not align with clinical practice and its \textit{do-no-harm} ethos. We recommend software solutions that would integrate into client care by: amplifying clinicians' expertise, revealing appropriate fabrication opportunities, and supporting adaptable fabrication.2019MHMegan Hofmann et al.Carnegie Mellon UniversityEarly Childhood Education TechnologyDesktop 3D Printing & Personal FabricationCircuit Making & Hardware PrototypingCHI