AudioXtend: Assisted Reality Visual Accompaniments for Audiobook Storytelling During Everyday Routine Tasks

Document Title

AudioXtend: Assisted Reality Visual Accompaniments for Audiobook Storytelling During Everyday Routine Tasks

Document Information

• Subject Area: Assisted reality technology, multitasking in daily routines, learning technologies
• Keywords: Smart glasses, Optical Head-Mounted Display (OHMD), Assisted Reality (aR), incidental learning, memory enhancement, visual storytelling, audiobook augmentation, Heads-Up Computing

Research Background and Problem

• What problems or challenges did the authors identify?

  • Audiobooks perform well in multitasking scenarios but show significantly lower learning and memory effectiveness compared to traditional text.
  • Audio, as an "invisible, intangible, and transient" medium, does not provide visual anchors to help listeners better retain and absorb content.
  • Improving the learning and storytelling experience of audiobooks without compromising users' everyday task efficiency remains an unresolved issue.

• Why is this issue important?

  • As multitasking becomes a norm in modern life, audio is increasingly becoming a critical medium for information consumption.
  • Optimizing memory and learning outcomes for audio content could significantly enhance users' learning efficiency and storytelling engagement during fragmented time.

• Research Motivation and Related Work

  • Existing studies have shown that combining multimedia (e.g., visual and audio) can significantly improve attention, memory, and comprehension, but how to effectively integrate visual augmentation into audiobook scenarios remains unclear.
  • Based on cognitive theories, such as the dual-coding theory, the combination of visual and auditory information can enhance information processing and storage capabilities.

Solution

• What methods or solutions did the authors propose?

  • The authors proposed a technology called AudioXtend, which uses Optical Head-Mounted Displays (OHMD) to present quickly scannable AI-generated visual materials as supplements to audiobook content.

• What are the innovative aspects of this solution?

  • Enhancing the audiobook experience using assisted reality technology to enrich content absorption.
  • AI-generated visual designs synchronize with audio content, allowing users to switch attention freely in multitasking scenarios without requiring prolonged focus.

• What are the implementation steps? What key technologies were used?

  • Preliminary Study (Study 1): Conducted experiments comparing memory effects and narrative engagement between audio-only and audio-visual enhancement.
  • Participatory Design Workshop: Investigated specific conditions for visual design (e.g., image style, simplicity, adaptability).
  • Field Study: Conducted a 3-day real-world user experience study in environments such as home, commuting, and daily tasks.

Research Outcomes

• What specific results were achieved?

  • Experiments showed that compared to audio-only, combining audio with AI-generated visuals improved immediate recall by 33.3% and recall after 7 days by 32.7%.
  • Users demonstrated significantly enhanced narrative comprehension and overall engagement.
  • Visual augmentations displayed via OHMD did not significantly impact the efficiency of primary tasks.

• What are its advantages compared to existing solutions?

  • Compared to traditional video-based learning methods, AudioXtend provides glanceable content displays that do not interfere with users' primary tasks.
  • The image design emphasizes simplicity and transparency, minimizing visual distractions.

• What were the experimental or evaluation results?

  • AudioXtend showed significant advantages in improving both immediate and long-term memory retention.
  • Users generally reported satisfaction with the experience, particularly in understanding content and maintaining focused attention.

• Limitations and Future Directions

  • Limitations:
    • The experiments focused only on specific narrative text types and a limited range of daily tasks, failing to cover diverse scenarios.
    • The smart glasses hardware used had technical and comfort limitations, with issues such as outdoor lighting and social acceptability being prominent.
    • The participant sample primarily consisted of young adults from academic settings, which may not be representative of a broader user base.
  • Suggested Future Directions:
    • Develop dynamic, adaptive user scenario models.
    • Provide more customizable interface options, such as visual styles, color schemes, and functional modules.
    • Extend the study duration to gain insights into the long-term use of new technologies and changes in user behavior.
    • Strengthen ethical reviews and ensure consistency in the style of AI-generated visual content.