Mapping the Wizards' Path: A Systematic Review of Wizard-of-Oz in HCI

Paper Title

Mapping the Wizards' Path: A Systematic Review of Wizard-of-Oz in HCI

Publication Info

• Topic area: Systematic review of Wizard-of-Oz (WoZ) practices in Human-Computer Interaction (HCI).
• Keywords: Wizard-of-Oz, HCI, prototyping, interaction design, systematic review, SIGCHI, Large Language Models, automation, user studies, methodological concerns.

Background and Problem

• Problem / challenge: Despite the long-standing use of the Wizard-of-Oz (WoZ) method in HCI, prior reflections on its applications, motivations, and methodological concerns are scattered and domain-specific, lacking a comprehensive synthesis.
• Significance: A systematic understanding of WoZ is critical to clarify its evolving role in prototyping future interactions, especially as automation and advanced technologies like Large Language Models (LLMs) blur the line between simulation and implementation.
• Motivation and related work: Early work on WoZ focused on natural language interfaces and robotics, with limited reviews such as Riek’s survey in Human-Robot Interaction (HRI). However, these reviews are outdated, narrow in scope, and do not address WoZ’s broader applications in contemporary HCI. This paper aims to fill this gap by systematically reviewing WoZ practices across SIGCHI venues from 1988 to 2025.

Solution

• Proposed approach: A systematic literature review of 194 WoZ studies in HCI, identifying application domains, wizard control types, motivations, and methodological concerns, and proposing a reciprocal evolution framework to contextualize WoZ’s role in prototyping.
• Novelty:
  1. First large-scale systematic review of WoZ in HCI, covering 194 papers from SIGCHI venues (1988–2025).
  2. Identification of ten application domains, five wizard control types, eight motivations, and five categories of concerns.
  3. Introduction of a reciprocal evolution framework to explain how WoZ and technology co-evolve.
  4. Contextualization of WoZ within LLM-based prototyping, highlighting emerging co-use patterns.
• Procedure and key techniques:
  • Systematic corpus development using ACM Digital Library and PRISMA methodology, resulting in 194 papers.
  • Deductive and inductive coding to classify application domains, control types, motivations, and concerns.
  • Analysis of trends, methodological trade-offs, and emerging patterns of WoZ-LLM integration.

Results

• Concrete findings:
  • WoZ has been applied in ten domains, including Work & Creativity Support (27.8%), HRI (18.0%), and XR/VR/AR (11.9%).
  • Five wizard control types identified: Communication Control (55.2%), System Logic Control (48.5%), Embodied & Environmental Action Control (47.9%), Pre-scripted Content Control (36.6%), and Wizard Collaboration & Mediation (16.0%).
  • Eight motivations for employing WoZ: Technical Limitation (45.9%), Alignment with Prior Work (16.5%), Adaptive Flexibility (14.4%), Psychological Realism (12.9%), Controlling Confounds (12.4%), Cost-Efficiency (12.4%), Safety & Ethical Constraints (5.2%), and Benchmark (3.6%).
  • Five concerns with WoZ: Temporal and Performance Constraints (13.4%), Wizard Variability and Bias (8.2%), Ecological Validity (7.2%), Deception and Ethics (2.6%), and Transparency and Reproducibility (1.0%).
• Advantage over baselines: The review consolidates fragmented knowledge, providing a comprehensive map of WoZ practices and a novel framework for understanding its evolving role alongside advanced technologies like LLMs.
• Experiments / evaluation: Analysis based on a systematically developed corpus of 194 papers, with coding reliability ensured through inter-rater reliability (IRR) metrics (e.g., domains = 0.89, control types = 0.81).
• Limitations and future work:
  • Potential exclusion of studies using alternative terms for WoZ.
  • Limited to SIGCHI venues, possibly missing insights from adjacent fields.
  • Small sample size for WoZ-LLM co-use patterns (N = 13), requiring further validation.
  • Future work includes expanding the corpus and refining the reciprocal evolution framework for AGI-era contexts.

Summary

This paper presents the first systematic review of Wizard-of-Oz (WoZ) practices in HCI, analyzing 194 papers from SIGCHI venues (1988–2025). It identifies ten application domains, five wizard control types, eight motivations, and five methodological concerns, offering a comprehensive map of WoZ practices. The proposed reciprocal evolution framework explains how WoZ and technology co-evolve, with WoZ shifting to simulate new plausible behaviors as technical capabilities advance. The study also explores emerging WoZ-LLM co-use patterns, highlighting complementary roles and challenges. These findings provide actionable guidelines for employing WoZ rigorously and responsibly, supporting its continued relevance in prototyping future interactions.