Effects of Visual Modality on Conversations with Interactive Digital Testimonies: Preparing for the Post-Witness Era

Research Background and Problem Statement

• What issues or challenges did the authors identify?
  The number of firsthand witnesses to major historical events (e.g., Holocaust survivors) is gradually decreasing, making the digital preservation of authentic conversational experiences increasingly important. However, there is a lack of empirical research on how different output modalities of Interactive Digital Testimonies (IDTs), such as audio, 2D video, and 3D video, impact user experience.

• Why is this issue important?
  The purpose of IDTs is to preserve and disseminate human historical experiences interactively over the long term. Understanding how different output modalities affect user engagement, perceived authenticity, and immersion is crucial for optimizing educational and commemorative experiences, especially in the "post-witness era."

• Research Motivation and Related Work:
  Early studies have shown that the authenticity and vividness of witness stories are critical for emotional connection and learning outcomes. However, much of the research on interactive virtual characters has focused on synthetic voice and visual agents, with limited exploration of IDTs based on real recordings. This study aims to fill this gap and provide empirical evidence for future design optimizations.

Proposed Solution

• What methods or solutions did the authors propose?
  The paper proposes a method to compare user experiences across different output modalities (audio, 2D video, 3D video) through two mixed-method studies:

  1. Comparing the effects of audio and 2D video modalities in both online and in-person settings (Study 1, 82 participants).
  2. Comparing the user experience differences between 2D video and 3D video modalities (Study 2, 51 participants).

• What is innovative about this solution?

  1. The study extends research on virtual human agents based on real recordings rather than synthetic interactive agents.
  2. It explores the potential advantages of the 3D modality and its impact on immersion and authenticity.
  3. It employs a combination of methods (questionnaires, qualitative interviews) and considers interactions across different environments and device setups.

• What are the implementation steps and key technologies used?

  1. Two experimental designs: the first is a 2×2 experimental design comparing "audio vs. 2D output" in "online vs. in-person" settings; the second is a user comparison experiment of "2D vs. 3D output."
  2. Audio processing utilized Natural Language Processing (NLP) services (e.g., Google Dialogflow) to analyze language intent and match relevant responses.
  3. Witness interaction content was provided in audio, 2D, or 3D recordings. Data collection included standardized questionnaires (e.g., User Experience Questionnaire UEQ-S) and open-ended interviews.

Research Findings

• What specific findings were obtained?

  • Audio vs. 2D Video: The 2D modality significantly enhanced user experience, immersion, and perceived authenticity. The audio modality was considered to lack visual social cues, resulting in lower engagement.
  • 2D Video vs. 3D Video: The 3D modality demonstrated superior performance in authenticity and emotional impact but was limited by issues such as eye strain and discomfort from 3D glasses.
  • Key findings include: visual presentation is a critical factor in the user experience of IDTs. High-quality 2D video often meets most user needs, while the additional costs of 3D modalities require further evaluation based on specific application contexts.

• What advantages does this solution have compared to existing ones?

  • This study focuses on real audio and video recordings, avoiding the "Uncanny Valley" effect often associated with synthetic agents.
  • It provides clear directions for improving future educational and commemorative designs, such as exploring glasses-free 3D display technologies and offering dynamic subtitles to enhance accessibility.

• What were the experimental or evaluation results?
  Study 1 revealed:

  • 2D video users found it easier to focus and formed stronger emotional connections with digital witnesses, particularly through specific nonverbal cues (e.g., gestures, facial expressions).
  • Online and in-person interactions differed: online users were more tolerant of technical inconveniences, while in-person users reported higher emotional impact.

  Study 2 revealed:

  • 3D video enhanced spatial depth perception and provided more details, but prolonged viewing led to increased physical discomfort, offsetting its advantages.
  • User preferences were nearly evenly split between 2D and 3D, indicating the need to tailor modalities to specific use cases and target audiences.

• Limitations and Future Directions:

  1. Current 3D technology implementations may limit its advantages (e.g., reliance on 3D glasses). Future research could explore glasses-free 3D technologies or display systems that support shared viewing experiences.
  2. This study primarily focused on adult participants. Future research should include middle school students to examine its applicability in classroom teaching.
  3. The content tested mainly involved Holocaust testimonies. Further research is needed to verify whether similar results hold for other topics or celebratory themes.

Through this study, the authors provide valuable empirical data and design optimization recommendations for the future development of IDTs, particularly in strategies for balancing visual display quality and user experience.