MIRAGE: Enabling Real-Time Automotive Mediated Reality

Paper Title

MIRAGE: Enabling Real-Time Automotive Mediated Reality

Publication Info

• Topic area: Real-time automotive mediated reality (AMR) for safety, trust, and interaction design in vehicles.
• Keywords: Automotive Mediated Reality, augmented reality, diminished reality, modified reality, real-time systems, object detection, segmentation, depth estimation, inpainting, human-computer interaction.

Background and Problem

• Problem / challenge: Existing research on Automotive Mediated Reality (AMR) is limited to simulated environments, lacking real-world evaluation due to technical and cost barriers. No prior work integrates augmented, diminished, and modified reality into a unified framework for real-vehicle use.
• Significance: Addressing real-world AMR evaluation is critical for ecological validity, improving safety, trust, and user experience in manual and automated driving scenarios.
• Motivation and related work: Prior studies have explored AR, DR, and ModR in isolation, often in controlled environments. Real-world deployment faces challenges like object detection, segmentation, and real-time processing. This paper builds on these efforts by introducing a holistic, real-time AMR system.

Solution

• Proposed approach: MIRAGE (Mediation Interface for Real-time Automotive Graphics and Effects), an open-source Unity-based tool for creating and evaluating AMR effects in real vehicles.
• Novelty:
  1. A unified framework supporting AR, DR, and ModR effects in real-time.
  2. Modular and runtime-customizable design for flexible prototyping.
  3. Integration of state-of-the-art computational models for object detection, segmentation, depth estimation, and inpainting.
  4. Real-world evaluation of AMR effects through an expert user study.
• Procedure and key techniques:
  • Use of RGB cameras and HMDs to simulate Windshield Displays (WSDs).
  • A modular pipeline processes camera input for semantic segmentation (YOLO11), depth estimation (DepthAnythingV2), and inpainting (MI-GAN).
  • Implementation of 15 AMR effects across AR, DR, and ModR categories.
  • Real-time performance optimization through parallel processing and configurable model update rates.

Results

• Concrete findings:
  • MIRAGE achieved real-time performance with up to 34 FPS on consumer-grade GPUs.
  • Expert users rated usability as average (SUS: 66.94) and identified technical limitations like latency and detection inaccuracies.
  • AR effects like "Outline" and "Bounding Box" were rated highest for technical reliability, while DR effects like "Remove" received mixed feedback due to inpainting artifacts.
• Advantage over baselines: MIRAGE bridges the sim-to-real gap by enabling real-world AMR evaluation, unlike prior simulator-based studies. Its modular design allows for flexible customization and integration of new effects.
• Experiments / evaluation:
  • Conducted an expert user study (N=9) with participants from academia and industry.
  • Participants tested MIRAGE during a 15-minute test drive, providing qualitative and quantitative feedback on usability, workload, and AMR effects.
  • Key use cases identified included safety enhancement, cognitive load management, and experiential improvement.
• Limitations and future work:
  • Current technical constraints include latency, misclassification, and lack of object permanence.
  • Future work will focus on improving depth estimation, adding contextual adaptation parameters, and addressing ethical concerns like selective reality manipulation.

Summary

MIRAGE is the first open-source tool to enable real-time Automotive Mediated Reality (AMR) in real vehicles, integrating AR, DR, and ModR effects. It supports flexible prototyping and evaluation of AMR concepts using state-of-the-art computational models. An expert user study highlighted its potential for improving safety, trust, and user experience, while also identifying technical and ethical challenges. MIRAGE lays the foundation for advancing AMR research and applications in real-world driving scenarios.