Prototyping Multimodal GenAI Real-Time Agents with Counterfactual Replays and Hybrid Wizard-of-Oz

Paper Title

Prototyping Multimodal GenAI Real-Time Agents with Counterfactual Replays and Hybrid Wizard-of-Oz

Publication Info

• Topic area: Prototyping and evaluation methods for multimodal generative AI real-time agents.
• Keywords: Multimodal GenAI, real-time agents, counterfactual replay, hybrid Wizard-of-Oz, prompt engineering, user-centered design, HCI, proactive systems, iterative prototyping, evaluation toolkit.

Background and Problem

• Problem / challenge: Existing prototyping and evaluation methods for multimodal generative AI systems are insufficient for handling the situational complexity, contextual immediacy, and real-time interactions required in domain-specific tasks.
• Significance: Addressing these challenges is critical for enabling the development of proactive, context-aware AI agents that can assist users effectively in real-world workflows.
• Motivation and related work: Previous methods, such as prompt engineering, prompt chaining, and benchmarking, focus on text-based or predefined tasks and lack support for multimodal, time-sensitive interactions. Tools like Wizard-of-Oz and replay enactments provide partial solutions but fail to address the unique challenges of real-time multimodal systems.

Solution

• Proposed approach: A novel prototyping pipeline combining Counterfactual Video Replay Prompting and a Hybrid Wizard-of-Oz method to iteratively design and refine multimodal GenAI agent behaviors.
• Novelty:
  1. Introduction of Counterfactual Video Replay Prompting for immersive, context-rich evaluation of prompt strategies using recorded user sessions.
  2. Adaptation of a Hybrid Wizard-of-Oz method for live user session testing, enabling real-time evaluation and refinement of prompts.
  3. Development of a modular, open-source Counterfactual Replay Toolkit for rapid prototyping of multimodal GenAI behaviors.
  4. Demonstration of iterative prompt decomposition and refinement for improving real-time agent performance.
• Procedure and key techniques:
  1. Use recorded user sessions to test and refine prompts via Counterfactual Video Replay Prompting.
  2. Transition to live user sessions with a Hybrid Wizard-of-Oz setup, allowing human oversight and intervention during real-time interactions.
  3. Iteratively restructure prompts into smaller, specialized modules (e.g., task-phase classification and message generation).
  4. Validate refinements through cycles of replay-based and live-session evaluations.

Results

• Concrete findings:
  • Splitting a single, lengthy system prompt into specialized prompts improved message relevance and alignment with user needs.
  • Separating task-phase classification from message generation reduced misclassifications and improved downstream support quality.
  • Second Hybrid Wizard-of-Oz cycle achieved higher design outcome scores (M = 4.4, SD = 0.5) compared to the first cycle (M = 1.6, SD = 0.9).
• Advantage over baselines:
  • Counterfactual replay provided richer situational context than tabular evaluations, enabling better assessment of prompt effectiveness.
  • Hybrid Wizard-of-Oz sessions revealed real-time misclassifications and allowed immediate corrections, which were not possible with recorded data alone.
• Experiments / evaluation:
  • Conducted iterative evaluations using 15 recorded user sessions and 10 live user sessions with mechanical design tasks in Autodesk Fusion360.
  • Measured design outcomes (e.g., structural soundness, clearance) and analyzed user interactions via video interaction analysis.
• Limitations and future work:
  • Limited generalizability due to focus on a single domain (mechanical design).
  • Scalability challenges due to reliance on human oversight in Hybrid Wizard-of-Oz sessions.
  • Potential wizard bias and reproducibility concerns in live-session evaluations.
  • Future work includes integrating methods into existing tools, extending to other domains, and automating intervention timing.

Summary

This paper introduces a novel prototyping pipeline for multimodal GenAI real-time agents, combining Counterfactual Video Replay Prompting and a Hybrid Wizard-of-Oz method. These approaches enabled iterative refinement of prompts and system behaviors, addressing challenges in real-time, context-sensitive interactions. The methods were validated through the development of SocraBot, a proactive design support agent, demonstrating improved alignment with user needs and higher-quality design outcomes. The open-source Counterfactual Replay Toolkit provides a practical resource for HCI researchers and practitioners to prototype and evaluate multimodal GenAI systems in diverse domains.