Overcoming Translation Delays: Towards Better Subtitle Design for Foreign Language Conversations in Extended Reality

Paper Title

Overcoming Translation Delays: Towards Better Subtitle Design for Foreign Language Conversations in Extended Reality

Publication Info

• Topic area: Subtitle interface design in XR environments for multilingual communication.
• Keywords: XR subtitles, translation delay, user experience, co-presence, speech attribution, emotion attribution, interface design, immersive environments, multilingual communication.

Background and Problem

• Problem / challenge: Translation delays in XR subtitle systems disrupt user experience, comprehension, and co-presence in multilingual conversations. Prior work has not systematically addressed how interface designs can mitigate these challenges under varying delay conditions.
• Significance: Addressing translation delays is crucial for enabling natural, effective cross-lingual communication in XR environments, where non-verbal cues and real-time interaction are critical.
• Motivation and related work: Existing XR subtitle designs focus on spatial and attention-aware interfaces but do not account for the impact of translation delays. Studies on 2D interfaces and audio-visual latency provide limited insights for XR contexts, where spatial and temporal dynamics differ. This paper aims to fill the gap by exploring how delay affects user experience and by proposing delay-resilient subtitle designs.

Solution

• Proposed approach: Development and evaluation of three novel XR subtitle interfaces (Floating Up Subtitles, Last-Two Subtitles, and Merged Subtitles) to mitigate the negative effects of translation delays, compared against a baseline Scrolling Transcript design.
• Novelty:
  1. Systematic investigation of translation delay effects (0s to 6s) on comprehension, co-presence, and user experience in XR environments.
  2. Design and evaluation of three innovative subtitle interfaces tailored to varying delay conditions.
  3. Proposal of actionable design guidelines for XR subtitle systems to address translation delays.
• Procedure and key techniques:
  • Study 1: Tested the impact of delays (0s, 1.5s, 3s, 4.5s, 6s) on comprehension, co-presence, and user preferences using a Scrolling Transcript interface in a VR-simulated meeting.
  • Study 2: Evaluated the three new interfaces against the baseline under delays of 1.5s, 3s, and 4.5s, focusing on performance, user experience, and preferences.
  • Metrics included gaze behavior, task accuracy (decision-making, speech attribution, emotion attribution), subjective ease (SEQ), co-presence scores, and user experience (UEQ-S).

Results

• Concrete findings:
  • Delays beyond 3.0s significantly reduced speech and emotion attribution accuracy, co-presence, and user experience.
  • Merged Subtitles achieved the highest emotion attribution accuracy (71.5%) and was rated best for co-presence and ease of use.
  • Scrolling Transcript was preferred at high delays (4.5s) for its complete textual record but scored lowest for co-presence and hedonic quality.
  • Last-Two Subtitles caused anxiety due to its ephemeral history, leading to poor user experience.
  • Floating Up Subtitles provided moderate performance but was criticized for distracting motion.
• Advantage over baselines:
  • Merged Subtitles reduced gaze time on subtitles, improved emotion attribution accuracy, and enhanced co-presence compared to the baseline.
  • At high delays, Scrolling Transcript ensured reliable comprehension but at the cost of social connection.
• Experiments / evaluation:
  • Study 1: 15 participants tested five delay levels in a VR meeting using Scrolling Transcript.
  • Study 2: 24 participants evaluated four interfaces under three delay levels, with metrics including accuracy, subjective ratings, and co-presence.
• Limitations and future work:
  • Limited to task-oriented dialogues with controlled context; future studies should explore casual, unstructured conversations.
  • Conducted in small groups (three avatars); scalability to larger groups and more realistic non-verbal behaviors needs testing.
  • Delay intervals sampled at 1.5s steps; finer-grained intervals (e.g., 0.5s) could provide more detailed insights.
  • Future systems should include context-aware, user-configurable interfaces and real-world deployments.

Summary

This paper investigates the impact of translation delays on comprehension, co-presence, and user experience in XR subtitle systems. Study 1 identified that delays over 3.0s disrupt speech and emotion attribution, while zero-delay conditions impose cognitive pressure. Study 2 evaluated three novel interfaces, finding Merged Subtitles most effective for balancing comprehension and co-presence. The findings highlight the need for adaptive, user-configurable subtitle systems that address varying delay conditions. This research provides actionable guidelines for designing XR subtitle interfaces to enhance cross-lingual communication in immersive environments.