To Recommend or Not to Recommend: Designing and Evaluating AI-Enabled Decision Support for Time-Critical Medical EventsAI-enabled decision-support systems aim to help medical providers rapidly make decisions with limited information during medical emergencies. A critical challenge in developing these systems is supporting providers in interpreting the system output to make optimal treatment decisions. In this study, we designed and evaluated an AI-enabled decision-support system to aid providers in treating patients with traumatic injuries. We first conducted user research with physicians to identify and design information types and AI outputs for a decision-support display. We then conducted an online experiment with 35 medical providers from six health systems to evaluate two human-AI interaction strategies: (1) AI information synthesis and (2) AI information and recommendations. We found that providers were more likely to make correct decisions when AI information and recommendations were provided compared to receiving no AI support. We also identified two socio-technical barriers to providing AI recommendations during time-critical medical events: (1) an accuracy-time trade-off in providing recommendations and (2) polarizing perceptions of recommendations between providers. We discuss three implications for developing AI-enabled decision support used in time-critical events, contributing to the limited research on human-AI interaction in this context.2025AMAngela Mastrianni et al.AI-Assisted HealthcareCSCW
Beyond Decision Making: Considering Collaboration and Agency in the Design of AI-Based Decision-Support Systems for Fast-Response Medical TeamsIn addition to aiding decision making, AI-based clinical decision-support systems may need to consider and support provider agency and collaboration between medical providers. We analyzed collaboration and agency within fast-response teams, identifying implications for designing decision-support systems that not only facilitate decision making, but also collaboration and agency. Using an Actor-Network Theory approach, we reviewed videos of 12 pediatric trauma resuscitations and conducted a secondary analysis of 27 interviews with trauma team members. We identified actants in trauma resuscitation, shifts in agency that can occur within fast-response teams during medical emergencies, and factors considered by providers when envisioning the design of decision-support systems. From our analysis, we propose implications for existing human-AI guidelines when designing AI systems for fast-response medical teams. We also highlight parallels between the introduction of clinical practice guidelines and the introduction of AI-based decision-support systems, suggesting that these systems may influence the training and "clinical gaze" of providers.2025AMAngela Mastrianni et al.Team Work Makes the Dream WorkCSCW
Understanding Personal Protective Equipment Use in Interdisciplinary Medical Settings: Design Considerations for Just-in-Time Interactive Compliance AlertsWe examine the use of personal protective equipment (PPE) in two interdisciplinary medical settings to inform the design of just-in-time alerts and reminders for correcting PPE noncompliance. We reviewed videos of 26 pediatric resuscitations occurring over the course of the COVID-19 pandemic at an urban pediatric teaching hospital. Through video review, we identified causes for PPE noncompliance, activities that were frequently performed without PPE, instances in which PPE was intentionally removed, and mechanisms by which healthcare providers corrected PPE noncompliance. We also interviewed 18 registered nurses working in the hospital’s emergency department (ED) and intensive care unit (ICU) to better understand observed PPE behaviors and practices. Our results suggest that alert design will require considering the urgency of correcting PPE noncompliance against the urgency of tasks being performed. We discuss our findings through the lens of the COM-B framework and conclude by exploring design opportunities for just-in-time alerts and reminders for prompting PPE noncompliance corrections in dynamic medical work.2025ASAleksandra Sarcevic et al.Privacy by Design & User ControlPasswords & AuthenticationPrivacy Perception & Decision-MakingDIS
Understanding Delay Awareness and Mitigation Mechanisms through an Iterative Design and Evaluation of a Prototype Alert System for Complex TeamworkAlmost half of the preventable deaths in emergency care can be associated with a medical delay. Understanding how clinicians experience delays can lead to improved alert designs to increase delay awareness and mitigation. In this paper, we present the findings from an iterative user-centered design process involving 48 clinicians to develop a prototype alert system for supporting delay awareness in complex medical teamwork such as trauma resuscitation. We used semi-structured interviews and card-sorting workshops to identify the most common delays and elicit design requirements for the prototype alert system. We then conducted a survey to refine the alert designs, followed by near-live, video-guided simulations to investigate clinicians’ reactions to the alerts. We contribute to CSCW by designing a prototype alert system to support delay awareness in time-critical, complex teamwork and identifying four mechanisms through which teams mitigate delays.2023KZKatherine Ann Zellner et al.TeamsCSCW
Supporting Awareness of Dynamic Data: Approaches to Designing and Capturing Data within Interactive Clinical Checklists Automatically integrating data within interactive clinical checklists allows for enhanced dynamic displays, while also providing information needed for checklist adaptation to the context of the medical event. In this mixed-methods study, we used user-centered design sessions with clinicians to design a checklist interface that automatically captures and displays dynamic patient data. We compared the manual and automatic checklist versions during video-guided simulation sessions, evaluating the effects of automatic capture on clinicians’ interactions with dynamic data and their situation awareness. Despite clinicians’ concerns that automatic data capture would affect situation awareness, we found no significant difference in awareness scores. Participants preferred the automatic version, highlighting its improved accuracy and completeness. From our findings, we propose a framework for capturing dynamic data and designing dynamic data interfaces within interactive checklists. We conclude by discussing barriers and design opportunities for supporting awareness of data trends through checklists.2023AMAngela Mastrianni et al.Medical & Scientific Data VisualizationTelemedicine & Remote Patient MonitoringDIS
A Speech-Based Model for Tracking the Progression of Activities in Extreme Action TeamworkDesigning computerized approaches to support complex teamwork requires an understanding of how activity-related information is relayed among team members. In this paper, we focus on verbal communication and describe a speech-based model that we developed for tracking activity progression during time-critical teamwork. We situated our study in the emergency medical domain of trauma resuscitation and transcribed speech from 104 audio recordings of actual resuscitations. Using the transcripts, we first studied the nature of speech during 34 clinically relevant activities. From this analysis, we identified 11 communicative events across three different stages of activity performance—before, during, and after. For each activity, we created sequential ordering of the communicative events using the concept of narrative schemas. The final speech-based model emerged by extracting and aggregating generalized aspects of the 34 schemas. We evaluated the model performance by using 17 new transcripts and found that the model reliably recognized an activity stage in 98% of activity-related conversation instances. We conclude by discussing these results, their implications for designing computerized approaches that support complex teamwork, and their generalizability to other safety-critical domains.2022SJSwathi Jagannath et al.Team Collaboration; Team CollaborationCSCW
Designing Interactive Alerts to Improve Recognition of Critical Events in Medical EmergenciesVital sign values during medical emergencies can help clinicians recognize and treat patients with life-threatening injuries. Identifying abnormal vital signs, however, is frequently delayed and the values may not be documented at all. In this mixed-methods study, we designed and evaluated a two-phased visual alert approach for a digital checklist in trauma resuscitation that informs users about undocumented vital signs. Using an interrupted time series analysis, we compared documentation in the periods before (two years) and after (four months) the introduction of the alerts. We found that introducing alerts led to an increase in documentation throughout the post-intervention period, with clinicians documenting vital signs earlier. Interviews with users and video review of cases showed that alerts were ineffective when clinicians engaged less with the checklist or set the checklist down to perform another activity. From these findings, we discuss approaches to designing alerts for dynamic team-based settings.2021AMAngela Mastrianni et al.Telemedicine & Remote Patient MonitoringDIS
Checklist Design Reconsidered: Understanding Checklist Compliance and Timing of InteractionsWe examine the association between user interactions with a checklist and task performance in a time-critical medical setting. By comparing 98 logs from a digital checklist for trauma resuscitation with activity logs generated by video review, we identified three non-compliant checklist use behaviors: failure to check items for completed tasks, falsely checking items when tasks were not performed, and inaccurately checking items for incomplete tasks. Using video review, we found that user perceptions of task completion were often misaligned with clinical practices that guided activity coding, thereby contributing to non-compliant check-offs. Our analysis of associations between different contexts and the timing of check-offs showed longer delays when (1) checklist users were absent during patient arrival, (2) patients had penetrating injuries, and (3) resuscitations were assigned to the highest acuity. We discuss opportunities for reconsidering checklist designs to reduce non-compliant checklist use.2020LKLeah Kulp et al.Drexel UniversityUser Research Methods (Interviews, Surveys, Observation)Prototyping & User TestingCHI
Temporal Rhythms and Patterns of Electronic Documentation in Time-Critical Medical WorkWe examine nursing documentation on a newly implemented electronic flowsheet in medical resuscitations to identify the temporal patterns of documentation and how the recorded information supported time-critical teamwork. To determine when the information was documented, we compared timestamps from 58 flowsheet logs to those of verbal communications derived from video review. We also drew on observations of 95 resuscitations to understand the behaviors of nurse documenters. We found that only 8% of the verbal reports were documented in near real-time (one minute within the verbal report), while 42% of reports were not documented in the electronic flowsheet. In addition, 38% were documented early (before the verbal report) and 12% were documented with a delay, ranging from one to 58 minutes after the report. Our study showed that the electronic flowsheet design posed many challenges for real-time documentation, leading to paper-based workarounds and the use of free-text fields on the flowsheet to visualize and keep track of time, and to communicate temporal information to the team. These findings suggest that documenters shape the temporal rhythms of not only their own work but also the rhythms of the electronic record and medical process. We discuss the implications of these rhythms for EHR redesign to support real-time documentation in high-risk, safety-critical settings.2019SJSwathi Jagannath et al.Drexel UniversityTelemedicine & Remote Patient MonitoringPrototyping & User TestingCHI
Comparing the Effects of Paper and Digital Checklists on Team Performance in Time-Critical WorkThis mixed-methods study examines the effects of a tablet-based checklist system on team performance during a dynamic and safety-critical process of trauma resuscitation. We compared team performance from 47 resuscitations that used a paper checklist to that from 47 cases with a digital checklist to determine if digitizing a checklist led to improvements in task completion rates and in how fast the tasks were initiated for 18 most critical assessment and treatment tasks. We also compared if the checklist compliance increased with the digital design. We found that using the digital checklist led to more frequent completions of the initial airway assessment task but fewer completions of ear and lower extremities exams. We did not observe any significant differences in time to task performance, but found increased compliance with the checklist. Although improvements in team performance with the digital checklist were minor, our findings are important because they showed no adverse effects as a result of the digital checklist introduction. We conclude by discussing the takeaways and implications of these results for effective digitization of medical work.2019LKLeah Kulp et al.Drexel UniversityTelemedicine & Remote Patient MonitoringPrototyping & User TestingCHI
Coordination Mechanisms for Self-Organized Work in an Emergency Communication CenterWe describe an observational study of work coordination in an emergency communication center, where a collocated team of communication specialists engages in complex activities of communicating with pre-hospital medical teams, and coordinating patient care and transport. Unlike teams with clearly defined work roles and team structures that were introduced to increase work efficiency and minimize redundancy, the team we studied lacks the role differentiation. To better understand how complex work is accomplished under these conditions, we conducted in-situ observations in the center’s control room and interviewed communication specialists. We found that communication specialists self-organized by using a mix of material and immaterial coordination mechanisms, including work schedules, computer systems, and tacit agreements to coordinate tasks. Using these findings, we then identified three features of self-organized, collocated and time-critical teamwork that require technology support: awareness of task ownership, task self-assignment, and informal team hierarchy. We conclude by discussing technology requirements to support these teamwork features.2018ZZZhan Zhang et al.Collaboration in Clinical ContextsCSCW