To Recommend or Not to Recommend: Designing and Evaluating AI-Enabled Decision Support for Time-Critical Medical EventsAI-enabled decision-support systems aim to help medical providers rapidly make decisions with limited information during medical emergencies. A critical challenge in developing these systems is supporting providers in interpreting the system output to make optimal treatment decisions. In this study, we designed and evaluated an AI-enabled decision-support system to aid providers in treating patients with traumatic injuries. We first conducted user research with physicians to identify and design information types and AI outputs for a decision-support display. We then conducted an online experiment with 35 medical providers from six health systems to evaluate two human-AI interaction strategies: (1) AI information synthesis and (2) AI information and recommendations. We found that providers were more likely to make correct decisions when AI information and recommendations were provided compared to receiving no AI support. We also identified two socio-technical barriers to providing AI recommendations during time-critical medical events: (1) an accuracy-time trade-off in providing recommendations and (2) polarizing perceptions of recommendations between providers. We discuss three implications for developing AI-enabled decision support used in time-critical events, contributing to the limited research on human-AI interaction in this context.2025AMAngela Mastrianni et al.AI-Assisted HealthcareCSCW
Understanding Personal Protective Equipment Use in Interdisciplinary Medical Settings: Design Considerations for Just-in-Time Interactive Compliance AlertsWe examine the use of personal protective equipment (PPE) in two interdisciplinary medical settings to inform the design of just-in-time alerts and reminders for correcting PPE noncompliance. We reviewed videos of 26 pediatric resuscitations occurring over the course of the COVID-19 pandemic at an urban pediatric teaching hospital. Through video review, we identified causes for PPE noncompliance, activities that were frequently performed without PPE, instances in which PPE was intentionally removed, and mechanisms by which healthcare providers corrected PPE noncompliance. We also interviewed 18 registered nurses working in the hospital’s emergency department (ED) and intensive care unit (ICU) to better understand observed PPE behaviors and practices. Our results suggest that alert design will require considering the urgency of correcting PPE noncompliance against the urgency of tasks being performed. We discuss our findings through the lens of the COM-B framework and conclude by exploring design opportunities for just-in-time alerts and reminders for prompting PPE noncompliance corrections in dynamic medical work.2025ASAleksandra Sarcevic et al.Privacy by Design & User ControlPasswords & AuthenticationPrivacy Perception & Decision-MakingDIS
Understanding Delay Awareness and Mitigation Mechanisms through an Iterative Design and Evaluation of a Prototype Alert System for Complex TeamworkAlmost half of the preventable deaths in emergency care can be associated with a medical delay. Understanding how clinicians experience delays can lead to improved alert designs to increase delay awareness and mitigation. In this paper, we present the findings from an iterative user-centered design process involving 48 clinicians to develop a prototype alert system for supporting delay awareness in complex medical teamwork such as trauma resuscitation. We used semi-structured interviews and card-sorting workshops to identify the most common delays and elicit design requirements for the prototype alert system. We then conducted a survey to refine the alert designs, followed by near-live, video-guided simulations to investigate clinicians’ reactions to the alerts. We contribute to CSCW by designing a prototype alert system to support delay awareness in time-critical, complex teamwork and identifying four mechanisms through which teams mitigate delays.2023KZKatherine Ann Zellner et al.TeamsCSCW
Supporting Awareness of Dynamic Data: Approaches to Designing and Capturing Data within Interactive Clinical Checklists Automatically integrating data within interactive clinical checklists allows for enhanced dynamic displays, while also providing information needed for checklist adaptation to the context of the medical event. In this mixed-methods study, we used user-centered design sessions with clinicians to design a checklist interface that automatically captures and displays dynamic patient data. We compared the manual and automatic checklist versions during video-guided simulation sessions, evaluating the effects of automatic capture on clinicians’ interactions with dynamic data and their situation awareness. Despite clinicians’ concerns that automatic data capture would affect situation awareness, we found no significant difference in awareness scores. Participants preferred the automatic version, highlighting its improved accuracy and completeness. From our findings, we propose a framework for capturing dynamic data and designing dynamic data interfaces within interactive checklists. We conclude by discussing barriers and design opportunities for supporting awareness of data trends through checklists.2023AMAngela Mastrianni et al.Medical & Scientific Data VisualizationTelemedicine & Remote Patient MonitoringDIS
A Speech-Based Model for Tracking the Progression of Activities in Extreme Action TeamworkDesigning computerized approaches to support complex teamwork requires an understanding of how activity-related information is relayed among team members. In this paper, we focus on verbal communication and describe a speech-based model that we developed for tracking activity progression during time-critical teamwork. We situated our study in the emergency medical domain of trauma resuscitation and transcribed speech from 104 audio recordings of actual resuscitations. Using the transcripts, we first studied the nature of speech during 34 clinically relevant activities. From this analysis, we identified 11 communicative events across three different stages of activity performance—before, during, and after. For each activity, we created sequential ordering of the communicative events using the concept of narrative schemas. The final speech-based model emerged by extracting and aggregating generalized aspects of the 34 schemas. We evaluated the model performance by using 17 new transcripts and found that the model reliably recognized an activity stage in 98% of activity-related conversation instances. We conclude by discussing these results, their implications for designing computerized approaches that support complex teamwork, and their generalizability to other safety-critical domains.2022SJSwathi Jagannath et al.Team Collaboration; Team CollaborationCSCW
Designing Interactive Alerts to Improve Recognition of Critical Events in Medical EmergenciesVital sign values during medical emergencies can help clinicians recognize and treat patients with life-threatening injuries. Identifying abnormal vital signs, however, is frequently delayed and the values may not be documented at all. In this mixed-methods study, we designed and evaluated a two-phased visual alert approach for a digital checklist in trauma resuscitation that informs users about undocumented vital signs. Using an interrupted time series analysis, we compared documentation in the periods before (two years) and after (four months) the introduction of the alerts. We found that introducing alerts led to an increase in documentation throughout the post-intervention period, with clinicians documenting vital signs earlier. Interviews with users and video review of cases showed that alerts were ineffective when clinicians engaged less with the checklist or set the checklist down to perform another activity. From these findings, we discuss approaches to designing alerts for dynamic team-based settings.2021AMAngela Mastrianni et al.Telemedicine & Remote Patient MonitoringDIS
Checklist Design Reconsidered: Understanding Checklist Compliance and Timing of InteractionsWe examine the association between user interactions with a checklist and task performance in a time-critical medical setting. By comparing 98 logs from a digital checklist for trauma resuscitation with activity logs generated by video review, we identified three non-compliant checklist use behaviors: failure to check items for completed tasks, falsely checking items when tasks were not performed, and inaccurately checking items for incomplete tasks. Using video review, we found that user perceptions of task completion were often misaligned with clinical practices that guided activity coding, thereby contributing to non-compliant check-offs. Our analysis of associations between different contexts and the timing of check-offs showed longer delays when (1) checklist users were absent during patient arrival, (2) patients had penetrating injuries, and (3) resuscitations were assigned to the highest acuity. We discuss opportunities for reconsidering checklist designs to reduce non-compliant checklist use.2020LKLeah Kulp et al.Drexel UniversityUser Research Methods (Interviews, Surveys, Observation)Prototyping & User TestingCHI
Comparing the Effects of Paper and Digital Checklists on Team Performance in Time-Critical WorkThis mixed-methods study examines the effects of a tablet-based checklist system on team performance during a dynamic and safety-critical process of trauma resuscitation. We compared team performance from 47 resuscitations that used a paper checklist to that from 47 cases with a digital checklist to determine if digitizing a checklist led to improvements in task completion rates and in how fast the tasks were initiated for 18 most critical assessment and treatment tasks. We also compared if the checklist compliance increased with the digital design. We found that using the digital checklist led to more frequent completions of the initial airway assessment task but fewer completions of ear and lower extremities exams. We did not observe any significant differences in time to task performance, but found increased compliance with the checklist. Although improvements in team performance with the digital checklist were minor, our findings are important because they showed no adverse effects as a result of the digital checklist introduction. We conclude by discussing the takeaways and implications of these results for effective digitization of medical work.2019LKLeah Kulp et al.Drexel UniversityTelemedicine & Remote Patient MonitoringPrototyping & User TestingCHI
Inspiring AWE: Transforming Clinic Waiting Rooms into Informal Learning Environments with Active Waiting EducationThis research explores patient education in pediatric hematology and oncology clinics. Based on interviews, observations, and a review of existing patient materials, we argue that education in clinic waiting rooms is in need of reform. We applied design principles from research in science museums along with tangible interaction techniques to create the Sickle Cell Station, an interactive learning experience about sickle cell disease. To evaluate the effectiveness of this design we observed approximately 580 participants in a pediatric hematology clinic waiting area in four different design conditions. These observations included detailed video analysis of 81 patients and their parents to understand their interaction and learning with the Sickle Cell Station. Our results show an engaging learning experience with relevant conversation, inquiry, and collaboration. We describe how patient engagement varied in the four design conditions and conclude with implications for new designs in the area of Active Waiting Education (AWE).2018ZLZeina Atrash Leong et al.Atraeleo, Northwestern UniversitySTEM Education & Science CommunicationSpecial Education TechnologyMuseum & Cultural Heritage DigitizationCHI