Human-AI Interaction for Time-Critical Sensemaking in Missing Persons Investigations

Paper Title

Human-AI Interaction for Time-Critical Sensemaking in Missing Persons Investigations

Publication Info

• Topic area: AI-driven tools for high-stakes sensemaking in missing persons investigations.
• Keywords: Human-AI interaction, missing persons, sensemaking, participatory design, large language models, data visualization, policing, high-stakes environments, summarization, information extraction.

Background and Problem

• Problem / challenge: Missing persons investigations require rapid synthesis of fragmented data, but current workflows are time-consuming and prone to information overload. Existing AI tools, particularly LLMs, face challenges such as hallucinations, lack of traceability, and limited domain adaptability, making them unreliable in high-stakes contexts.
• Significance: Enhancing efficiency in missing persons investigations can save lives, as delays in synthesizing critical information can hinder search efforts.
• Motivation and related work: Prior research has explored AI applications in policing, such as facial recognition and multimodal data synthesis, but few studies address language processing for investigative synthesis in missing persons contexts. Existing tools often neglect user needs, leading to poor adoption and limited utility.

Solution

• Proposed approach: A hybrid AI-aided system combining LLM-based summarization and extraction with rule-based methods and data-driven visualizations, designed iteratively with input from Police Scotland officers.
• Novelty:
  1. Identification of five high-value information categories for missing persons investigations: vulnerabilities, current incident circumstances, association networks, locations, and disappearance patterns.
  2. Development of a hybrid system that mitigates hallucination risks through source linking, rule-based methods, and data-driven visualizations.
  3. Participatory design process involving police officers to ensure the system aligns with real-world workflows and needs.
  4. Evaluation of LLM feasibility using synthetic data to assess hallucination rates and extraction accuracy.
• Procedure and key techniques:
  1. Conducted formative studies with Police Scotland officers to identify user needs and workflow requirements.
  2. Developed and refined three prototype concepts along design axes (modality, abstraction level, fidelity).
  3. Evaluated prototypes through workshops and surveys with officers, incorporating feedback into a final design.
  4. Assessed LLM performance using synthetic data to evaluate hallucination rates and extraction reliability.

Results

• Concrete findings:
  • Final prototype rated highly by officers, with 16 out of 17 participants stating it would improve current practices.
  • LLM-based entity extraction achieved high reliability (e.g., 87% match rate for association networks), but abstractive tasks like pattern extraction showed higher hallucination rates.
  • Visual elements (e.g., maps, timelines) were particularly valued for rapid decision-making.
• Advantage over baselines:
  • Hybrid approach reduced hallucination risks compared to LLM-only solutions.
  • Source linking ensured traceability and accountability, addressing critical concerns in high-stakes environments.
• Experiments / evaluation:
  • Formative study with 6 officers identified key requirements.
  • Concept refinement workshop with 2 Police Search Advisors evaluated three prototypes.
  • Final prototype tested via an online survey with 17 officers, with feedback on usefulness and completeness.
  • Synthetic data evaluation quantified hallucination rates and extraction accuracy across interface sections.
• Limitations and future work:
  • Limited participant pool and access to real data due to privacy constraints.
  • Evaluation relied on perceived usefulness rather than in-situ testing during real investigations.
  • Future work could explore multi-modal data integration (e.g., transport, social media) and extend the tool to other jurisdictions.

Summary

This paper presents a hybrid AI-aided system for missing persons investigations, developed through a participatory design process with Police Scotland. The system combines LLM-based summarization and extraction with rule-based methods and visualizations to address user needs while mitigating hallucination risks. Evaluation showed high perceived usefulness, particularly for visual elements and entity extraction tasks, though abstractive tasks like pattern recognition remain challenging. The study highlights the importance of user-centered design, source linking, and lightweight, local solutions in high-stakes contexts. Future work could expand the system's scope and evaluate its real-world impact.