Exploring Interface Design of MR Translation System for Everyday Interaction

Paper Title

Exploring Interface Design of MR Translation System for Everyday Interaction

Publication Info

• Topic area: Mixed Reality (MR) translation systems and their interface designs for real-world usability.
• Keywords: Mixed Reality, translation systems, interface design, user experience, cognitive load, real-time translation, Wizard-of-Oz, usability, spatial alignment, generative AI.

Background and Problem

• Problem / challenge: Existing MR translation systems primarily rely on overlay-based designs, which can cause visual discomfort, cognitive load, and usability issues in complex or interactive tasks. The effects of alternative interface designs remain underexplored.
• Significance: MR translation systems have the potential to alleviate language barriers and enhance multilingual communication in everyday contexts, but their usability and effectiveness depend on well-designed interfaces.
• Motivation and related work: Previous research has focused on single overlay-based designs and simple reading tasks, neglecting diverse interface paradigms and complex real-world scenarios. This paper addresses these gaps by systematically comparing multiple MR translation interface designs.

Solution

• Proposed approach: Systematic comparison of three MR translation interface designs—Backgrounded, Widget, and Integrated—evaluated through a Wizard-of-Oz user study in both passive and interactive tasks.
• Novelty:
  1. Comparative evaluation of three distinct MR translation interface designs.
  2. Assessment of usability and user experience in realistic, everyday scenarios with varying content complexity.
  3. Identification of design recommendations and future research directions for MR translation systems.
• Procedure and key techniques:
  • Three interface designs:
    • Backgrounded: Overlaying translations with an opaque background.
    • Widget: Displaying translations in a separate, movable virtual widget.
    • Integrated: Seamlessly replacing original text with translations using generative AI.
  • Wizard-of-Oz study with 24 participants performing non-interactive (reading) and interactive (physical interaction) tasks.
  • Metrics: Readability, task completion time, error rates, cognitive workload (NASA-TLX), usability (SUS), and user preferences.

Results

• Concrete findings:
  • Integrated: Most preferred design overall, offering seamless spatial alignment but hindered by typographical errors and latency.
  • Backgrounded: Effective for static tasks but caused minor visual interference during physical interactions.
  • Widget: Least effective due to cross-referencing demands and low readability, though it performed better in low-density interactive tasks.
  • Integrated had the highest SUS score for non-interactive tasks (71.98), while Backgrounded had the highest activation count for interactive tasks (M = 12.54).
• Advantage over baselines:
  • Integrated and Backgrounded outperformed Widget in readability, task completion time, and user satisfaction for most scenarios.
  • Integrated avoided visual obstruction but introduced cognitive effort due to rendering errors.
• Experiments / evaluation:
  • Tasks: Non-interactive (reading building directories and menu boards) and interactive (using kiosks for certificates and food ordering).
  • Controlled lab setup with Meta Quest 3 HMD and LG touchscreen monitor.
  • Statistical analysis of usability metrics and qualitative feedback from participants.
• Limitations and future work:
  • Limitations: Controlled lab setting, homogeneous participant pool, and simulated generative AI latency.
  • Future work: Explore dynamic overlays, improve generative AI accuracy, address ecological validity, and examine diverse user populations and real-world scenarios.

Summary

This study systematically compared three MR translation interface designs—Backgrounded, Widget, and Integrated—through a Wizard-of-Oz experiment involving 24 participants. Integrated was the most preferred design overall due to its seamless spatial alignment, though it faced challenges with rendering errors and latency. Backgrounded provided effective spatial alignment but caused minor visual interference during interactive tasks. Widget was the least effective, particularly for high-density content, due to cross-referencing demands. The findings highlight the importance of spatial alignment, legibility, and interaction-aware design in MR translation systems. Design recommendations and future research directions emphasize improving generative AI accuracy, dynamic overlays, and ecological validity for real-world applications.