Designing Multi-Robot Ground Video Sensemaking with Public Safety Professionals

Paper Title

Designing Multi-Robot Ground Video Sensemaking with Public Safety Professionals

Publication Info

• Topic area: Human-AI collaboration for multi-robot video analysis in public safety.
• Keywords: Multi-robot systems, public safety, video sensemaking, anomaly detection, human-AI collaboration, situational awareness, descriptor-based search, ground robots, computer vision, ethical AI.

Background and Problem

• Problem / challenge: Current public safety workflows rely on manual, labor-intensive video analysis, which is inefficient for multi-robot systems. Existing anomaly detection models and video tools are misaligned with the operational needs of public safety professionals.
• Significance: Enhancing video sensemaking in public safety can improve situational awareness, reduce manual workload, and enable scalable operations despite staffing constraints.
• Motivation and related work: Prior research in HCI and computer vision has explored video interaction tools and anomaly detection, but these systems are not tailored to multi-robot, mobile video streams or grounded in public safety professionals’ practices. Gaps include the lack of datasets, event taxonomies, and systems designed for multi-robot video workflows.

Solution

• Proposed approach: Multi-Robot Video Sensemaking System (MRVS), a human-AI collaborative tool combining a multimodal LLM back-end with a user-centered front-end for analyzing multi-robot video streams.
• Novelty:
  1. Introduction of a testbed environment with a taxonomy of 38 events of interest (EoIs), a public dataset of ground-robot videos, and six design requirements (DRs).
  2. Development of MRVS, integrating AI-driven video analysis with interactive features for situational awareness and collaboration.
  3. Evaluation of MRVS’s back-end and front-end performance through benchmarking and expert reviews.
  4. Identification of ethical, practical, and societal considerations for deploying MRVS-like systems.
• Procedure and key techniques:
  1. Co-design of EoIs and DRs through surveys and interviews with public safety professionals.
  2. Creation of a 20-video dataset featuring scripted patrol scenarios with 156 events across day and night conditions.
  3. Development of MRVS with features like Video Debrief, Situational Overview, Descriptor-Based Search, and Group Workspace.
  4. Evaluation through algorithm benchmarking and expert reviews with nine public safety professionals.

Results

• Concrete findings:
  • MRVS achieved F1 scores of 0.497 (day), 0.540 (night), and 0.519 (overall), outperforming HolmesVAD and Gemini 2.0 in anomaly detection.
  • Professionals reported reduced manual workload, faster investigations, and improved situational awareness.
• Advantage over baselines:
  • MRVS demonstrated higher recall (0.663 overall) and better handling of nighttime conditions compared to baselines.
  • The system’s descriptor-based search and AI-generated debriefs were praised for efficiency and usability.
• Experiments / evaluation:
  • Algorithm evaluation on 20 videos (470 minutes) compared MRVS with HolmesVAD and Gemini 2.0.
  • Expert review with nine professionals assessed MRVS’s usability, functionality, and integration into workflows.
• Limitations and future work:
  • Simulated patrol scenarios may not fully capture real-world edge cases or adversarial behaviors.
  • Limited generalizability due to single jurisdiction and campus environment.
  • Future work includes expanding EoIs, datasets, and deployment to diverse jurisdictions and modalities.

Summary

This paper introduces MRVS, a human-AI collaborative system for multi-robot video sensemaking in public safety. By combining a multimodal LLM back-end with a user-centered front-end, MRVS enhances situational awareness, reduces manual workload, and supports collaborative workflows. Evaluations show significant improvements in anomaly detection and usability compared to baselines. The work also provides a reusable testbed environment and identifies ethical and practical considerations for deploying MRVS-like systems. Future research should focus on broader deployments, additional modalities, and long-term adaptation to operational contexts.