"What do I do now?": Spontaneous Human Responses to Robot Effectiveness and Efficiency Malfunctions in Collaborative Robotics

Paper Title

'What do I do now?': Spontaneous Human Responses to Robot Effectiveness and Efficiency Malfunctions in Collaborative Robotics

Publication Info

• Topic area: Human responses to robot malfunctions in collaborative settings.
• Keywords: Human–robot collaboration, robot malfunctions, situational awareness, task-oriented responses, spontaneous behavior, efficiency malfunctions, effectiveness malfunctions, workload, confusion, transparency.

Background and Problem

• Problem / challenge: While prior research has focused on the system side of robot malfunctions (e.g., detection, recovery, and trust maintenance), little is known about how humans spontaneously respond to malfunctions in real-time and whether their actions support or hinder collaboration.
• Significance: Understanding human responses to malfunctions is critical for designing collaborative robots that can adapt to diverse human behaviors, enhance collaboration, and maintain productivity during disruptions.
• Motivation and related work: Previous studies have explored multimodal signals (e.g., gaze, vocalizations) during robot failures but have not systematically linked these to task-oriented appropriateness. This study addresses this gap by analyzing unscripted human responses to robot malfunctions in terms of situational awareness (SA) and task alignment.

Solution

• Proposed approach: A within-subjects study observing 65 participants collaborating with a robot during normal operation and induced malfunctions (effectiveness and efficiency types).
• Novelty:
  1. Empirical analysis of unscripted vocal and action responses to robot malfunctions.
  2. Linking SA and task-oriented appropriateness to highlight behavioral variability.
  3. Design implications for collaborative robots emphasizing transparency, explainability, and personalized support.
• Procedure and key techniques:
  • Participants performed a quality control and repair task alongside a robot across three blocks: normal operation (Block 1), malfunction (Block 2), and return to normal operation (Block 3).
  • Malfunctions were categorized as effectiveness (e.g., faulty items sent to packaging) or efficiency (e.g., functional items sent for repair).
  • Data collected included unscripted vocal and action responses, task performance, and subjective ratings of workload, confusion, and malfunction severity.
  • Responses were coded for SA (perception, comprehension, projection) and task-oriented appropriateness (productive, unproductive, counterproductive).

Results

• Concrete findings:
  • 164 of 167 unscripted responses occurred during malfunctions, with efficiency malfunctions eliciting more responses (126) than effectiveness malfunctions (36).
  • Responses were categorized as 89 productive, 38 unproductive, and 37 counterproductive.
  • Effectiveness malfunctions were rated as more severe (M = 80.56) than efficiency malfunctions (M = 47.05).
  • Workload decreased across blocks, but mental demand, frustration, and confusion peaked during malfunctions.
• Advantage over baselines:
  • Demonstrated that SA was nearly universal during malfunctions, but task-oriented appropriateness varied significantly.
  • Efficiency malfunctions prompted more productive responses due to their actionability, while effectiveness malfunctions often led to inaction or counterproductive behaviors.
• Experiments / evaluation:
  • Conducted with 65 participants in a controlled laboratory setting.
  • Data analyzed using repeated measures ANOVA and coding of unscripted responses.
  • Subjective measures included NASA-TLX workload ratings and confusion/severity scales.
• Limitations and future work:
  • Limited ecological validity due to simplified task and scripted malfunctions.
  • Focused on first-time reactions without training or longitudinal adaptation.
  • Future work should explore richer contexts, more diverse malfunction types, and real-time severity assessments.

Summary

This study investigated unscripted human responses to robot malfunctions in a collaborative task, analyzing situational awareness and task-oriented appropriateness. While participants universally detected malfunctions, their responses ranged from productive to counterproductive, influenced by the type and actionability of the malfunction. Efficiency malfunctions elicited more productive responses, whereas effectiveness malfunctions were perceived as more severe but often led to inaction. Subjective ratings revealed increased mental demand, frustration, and confusion during malfunctions. These findings highlight the need for collaborative robots to bridge the gap between awareness and action by providing transparency, explainability, and context-sensitive support.