A Comparative Analysis of Information Gathering by Chatbots, Questionnaires, and Humans in Clinical Pre-ConsultationInformation gathering is an important capability that allows chatbots to understand and respond to users' needs, yet the effectiveness of LLM-powered chatbots at this task remains underexplored. Our work investigates this question in the context of clinical pre-consultation, wherein patients provide information to an intermediary before meeting with a physician to facilitate communication and reduce consultation inefficiencies. We conducted a study at a walk-in clinic with 45 patients who interacted with one of three conversational agents: a chatbot, a questionnaire, and a Wizard-of-Oz. We analyzed patients' messages using metrics adapted from Grice's maxims to assess the quality of information gathered at each conversation turn. We found that the Wizard and LLM were more successful than the questionnaire because they modified questions and asked follow-ups when participants provided unsatisfactory answers. However, the LLM did not ask nearly as many follow-up questions as the Wizard, particularly when participants provided unclear answers.2025BLBrenna Li et al.University of Toronto, Computer ScienceMid-Air Haptics (Ultrasonic)Conversational ChatbotsHuman-LLM CollaborationCHI
Beyond the Waiting Room: Patient's Perspectives on the Conversational Nuances of Pre-Consultation ChatbotsPre-consultation serves as a critical information exchange between healthcare providers and patients, streamlining visits and supporting patient-centered care. Human-led pre-consultations offer many benefits, yet they require significant time and energy from clinical staff. In this work, we identify design goals for pre-consultation chatbots given their potential to carry out human-like conversations and autonomously adapt their line of questioning. We conducted a study with 33 walk-in clinic patients to elicit design considerations for pre-consultation chatbots. Participants were exposed to one of two study conditions: an LLM-powered AI agent and a Wizard-of-Oz agent simulated by medical professionals. Our study found that both conditions were equally well-received and demonstrated comparable conversational capabilities. However, the extent of the follow-up questions and the amount of empathy impacted the chatbot's perceived thoroughness and sincerity. Patients also highlighted the importance of setting expectations for the chatbot before and after the pre-consultation experience.2024BLBrenna Li et al.University of TorontoConversational ChatbotsHuman-LLM CollaborationMental Health Apps & Online Support CommunitiesCHI
Functional Design Requirements to Facilitate Menstrual Health Data ExplorationMenstrual trackers currently lack the affordances required to help individuals achieve their goals beyond menstrual event predictions and symptom logging. Taking an initial step towards this aspiration, we propose, validate, and refine five functional design requirements for future interface designs that facilitate menstrual data exploration. We interviewed 30 individuals who menstruate and collected their feedback on the practical application of these requirements. To elicit ideas and impressions, we designed two proof-of-concept interfaces to use as design probes with similar core functionalities but different presentations of phase timing predictions and signal arrangement. Our analysis revealed participants' feedback regarding the presentation of predictions for menstrual-related events, the visualization of future signal patterns, personalization abilities for viewing signals relevant to their menstrual experience, the availability of resources to understand the underlying biological connections between signals, and the ability to compare multiple cycles side-by-side with context.2024GLGeorgianna Lin et al.University of TorontoReproductive & Women's HealthDiet Tracking & Nutrition ManagementCHI
Behind the Pup-ularity Curtain: Understanding the Motivations, Challenges, and Work Performed in Creating and Managing Pet Influencer AccountsCreating dedicated accounts to post users’ pet content is a growing trend on Instagram. While these account owners derive joy from this pursuit, they may also struggle with criticisms and challenges. Yet, there remains a knowledge gap on how pet account owners manage their pets' online presence and navigate these obstacles successfully. Drawing from interviews with 21 Instagram pet account owners, we uncover the motivations behind pet account creation, spanning personal, altruistic, and commercial goals. We learn about the strategies employed for crafting their pets' online identities and personas, as well as the challenges faced by both owners and their pets in navigating the complexities of digital identity management. We discuss the evolving dynamics between humans and their pets, positioning pet identity cultivation as a form of collaborative work, akin to the ``third shift'', highlighting the need to design interfaces that support this unique identity management process.2024SYSuhyeon Yoo et al.University of TorontoContent Moderation & Platform GovernanceOnline Identity & Self-PresentationCHI
Understanding tensions in music accessibility through song signing for and with d/Deaf and Non-d/Deaf personsSong signing is a method practiced by people who are d/Deaf and non-d/Deaf individuals to visually represent music and make music accessible through sign language and body movements. Although there is growing interest in song signing, there is a lack of understanding on what d/Deaf people value about song signing and how to make song signing productions they would consider acceptable. We conducted semi-structured interviews with 12 d/Deaf participants to gain a deeper understanding of what they value in music and song signing. We then interviewed 14 song signers to understand their experiences and processes in creating song signing performances. From this study, we identify three complex, interrelated layers of the song signing creation process and discuss how they can be supported and completed to potentially bridge the cultural divide between the d/Deaf and non-d/Deaf audiences and guide more culturally responsive creation of music.2023SYSuhyeon Yoo et al.University of TorontoDeaf & Hard-of-Hearing Support (Captions, Sign Language, Vibration)Programming Education & Computational ThinkingCHI
Identifying Multimodal Context Awareness Requirements for Supporting User Interaction with Procedural VideosFollowing along how-to videos requires alternating focus between understanding procedural video instructions and performing them. Examining how to support these continuous context switches for the user has been largely unexplored. In this paper, we describe a user study with thirty participants who performed an hour-long cooking task while interacting with a wizard-of-oz hands-free interactive system that is aware of both their cooking progress and environment contexts. Through analysis of the session scripts, we identify a dichotomy between participant query differences and workflow alignment similarities, under-studied interactions that require AI functionality beyond video navigation alone, and queries that call for multimodal sensing of a user’s environment. By understanding the assistant experience through the participants’ interactions, we identify design implications for a smart assistant that can discern a user’s task completion flow and personal characteristics, accommodate requests within and external to the task domain, and support nonvoice-based queries.2023GLGeorgianna Lin et al.University of Toronto, University of TorontoVoice User Interface (VUI) DesignContext-Aware ComputingCHI
Constraints and Workarounds to Support Clinical Consultations in Synchronous Text-based PlatformsMedical consultations over synchronous text-based platforms are becoming increasingly popular for virtual care, yet little is known about how physicians translate their training to this healthcare medium. We report the constraints, workarounds, and opportunities highlighted by eight primary care physicians who used such a platform in simulated medical scenarios with standardized patients. We found that due to the perceived inefficiency of communicating over text, the physicians made subconscious use of double-barreled questions and action multiplexing to streamline the conversation. In addition, the physicians overcame the lack of missing verbal and visual cues by adding explicit messages to convey empathy and active listening. We also identify several affordances of text-based platforms, such as the ability for users to reference the conversation history and for patients to feel a sense of privacy during sensitive disclosure. From these findings, we propose design opportunities for how future synchronous text-based platforms can better support medical consultations.2023BLBrenna Li et al.University of TorontoMid-Air Haptics (Ultrasonic)Mental Health Apps & Online Support CommunitiesCHI
An Exploration of Captioning Practices and Challenges of Individual Content Creators on YouTube for People with Hearing ImpairmentsDeaf and Hard-of-Hearing (DHH) audiences have long complained about caption qualities for many online videos created by individual content creators on video-sharing platforms (e.g., YouTube). However, there lack explorations of practices, challenges, and perceptions of online video captions from the perspectives of both individual content creators and DHH audiences. In this work, we first explore DHH audiences' feedback on and reactions to YouTube video captions through interviews with 13 DHH individuals, and uncover DHH audiences' experiences, challenges, and perceptions on watching videos created by individual content creators (e.g., manually added caption tags could create additional confidence and trust in caption qualities for DHH audiences). We then discover individual content creators' practices, challenges, and perceptions on captioning their videos (e.g., back-captioning problems) by conducting a YouTube video analysis with 189 captioning-related YouTube videos, followed by a survey with 62 individual content creators. Overall, our findings provide an in-depth understanding of captions generated by individual content creators and bridge the knowledge gap mutually between content creators and DHH audiences on captions.2022FLFranklin Mingzhe Li et al.Accessibility; AccessibilityCSCW
Designing for Relational Maintenance: New Directions for AAC ResearchAAC research has traditionally focused on input speed, leaving higher-level communication goals such as relational maintenance under-explored. Through semi-structured interviews with AAC users with motor and speech impairments and their primary family caregivers, we offer a nuanced understanding of AAC’s roles in maintaining close relationships. Our inductive analysis reveals emerging themes including how AAC users and their partners share the physical and mental workload to overcome communication barriers in complex situations. Our deductive application of the Relational Maintenance Strategies framework exposes the efforts made and the challenges encountered in managing social engagements, providing mutual support, and decoding implicit expressions. From these insights, we propose novel research directions for better supporting maintenance strategies and social purposes of communication, including notably mediating relational tensions, leveraging empowerment and identity, and supporting interactions for social closeness and etiquette, which we hope will motivate discussion in HCI communities on expanding AAC research space.2022JDJiamin Dai et al.McGill UniversityCognitive Impairment & Neurodiversity (Autism, ADHD, Dyslexia)Augmentative & Alternative Communication (AAC)CHI
Automating Clinical Documentation with Digital Scribes: Understanding the Impact on PhysiciansRecently, digital scribe systems have been gaining popularity as a possible work-around solution to the Electronic Medical Record (EMR) documentation burden that affects many physicians. The proposed system would automate the clinical summary physicians take by capturing and extracting the patient-physician conversation during the consultation. While promising in concept, how this system would apply to real-world use and its limitations are still not well understood. To examine these issues, we designed a digital scribe prototype to generate notes of different qualities ranging from the reality of current state-of-the-art technology to the potential of future implementations. We conducted a "Wizard of Oz" study with 24 primary care physicians using our digital scribe prototype in 4 simulated medical encounters followed by a semi-structured interview. This exploratory study provides an understanding of physicians' interaction with digitally scribed notes, their perceptions on note quality, their perceived workflow impact and several directions for improvements.2021BLBrenna Li et al.University of TorontoHuman-LLM CollaborationAI-Assisted Decision-Making & AutomationCHI
"I Choose Assistive Devices That Save My Face" A Study on Perceptions of Accessibility and Assistive Technology Use Conducted in ChinaDespite the potential benefits of assistive technologies (ATs) for people with various disabilities, only around 7% of Chinese with disabilities have had an opportunity to use ATs. Even for those who have used ATs, the abandonment rate was high. Although China has the world's largest population with disabilities, prior research exploring how ATs are used and perceived, and why ATs are abandoned have been conducted primarily in North America and Europe. In this paper, we present an interview study conducted in China with 26 people with various disabilities to understand their practices, challenges, perceptions, and misperceptions of using ATs. From the study, we learned about factors that influence AT adoption practices (e.g., misuse of accessible infrastructure, issues with replicating existing commercial ATs), challenges using ATs in social interactions (e.g., Chinese stigma), and misperceptions about ATs (e.g., ATs should overcome inaccessible social infrastructures). Informed by the findings, we derive a set of design considerations to bridge the existing gaps in AT design (e.g., manual vs. electronic ATs) and to improve ATs' social acceptability in China.2021FLFranklin Mingzhe Li et al.Carnegie Mellon University, University of TorontoCognitive Impairment & Neurodiversity (Autism, ADHD, Dyslexia)Universal & Inclusive DesignSpecial Education TechnologyCHI