Towards Understanding the Design of Mixed Reality Systems to Enrich the Beverage Experience

Paper Title

Towards Understanding the Design of Mixed Reality Systems to Enrich the Beverage Experience

Publication Info

• Topic area: Human-Food Interaction (HFI) and immersive technology in multisensory beverage experiences.
• Keywords: Mixed Reality, Human-Food Interaction, bubble tea, flavor perception, multisensory design, adaptive systems, large language models, virtual environments, sensory engagement, personalization.

Background and Problem

• Problem / challenge: Existing Human-Food Interaction (HFI) systems are predominantly static and designer-driven, lacking adaptiveness and opportunities for embodied actions that support meaningful participant involvement. Little attention has been paid to how natural language input can shape multisensory responses in immersive environments.
• Significance: Multisensory design in beverage experiences could enhance sensory engagement, personalize flavor perception, and create playful and socially enriched consumption moments. This has implications for gastronomy, health, and interactive systems design.
• Motivation and related work: Prior research has explored static multisensory systems (e.g., MetaCookie+, Vocktail) and immersive VR environments but has not integrated dynamic, adaptive systems that respond to user input in real time. Advances in large language models (LLMs) offer opportunities for adaptive personalization, but their application to playful beverage experiences remains unexplored.

Solution

• Proposed approach: XTea, a Mixed Reality (MR) system combining physical drinking actions with adaptive virtual environments modulated by natural language input.
• Novelty:
  1. Introduction of XTea, a multisensory MR system integrating visual, auditory, and physical components for adaptive beverage experiences.
  2. Thematic findings on sensory engagement, personalized flavor modulation, and social presence in immersive environments.
  3. Four design strategies for multisensory beverage experiences, emphasizing dynamic visual cues, user agency, and social dimensions.
• Procedure and key techniques:
  • Participants interact with a custom-designed XTea cup and a virtual environment called “Flavor Drift” via a VR headset.
  • The system adapts visual, auditory, and geometric features based on spoken input, processed by GPT-4.5-preview.
  • Adaptive parameters include bloom intensity, lighting color, geometry quantity, environmental dynamics, and music selection.

Results

• Concrete findings:
  • Salient visual changes (e.g., color shifts) intensified sensory engagement, while subtle changes were often unnoticed.
  • Voice-based interaction enhanced participants’ sense of agency and personalization, transforming drinking into a co-created experience.
  • Narrative cues and virtual companions fostered purpose and social presence during the drinking experience.
• Advantage over baselines: XTea introduced dynamic adaptiveness and participant-driven sensory modulation, contrasting with static, designer-driven systems in prior HFI work.
• Experiments / evaluation:
  • Study with 12 bubble tea enthusiasts aged 19–36 years.
  • Participants experienced adaptive virtual environments while consuming personalized bubble tea.
  • Data collected through thematic analysis of interviews and behavioral observations.
• Limitations and future work:
  • Controlled lab setting limits ecological validity; future studies should explore real-world contexts.
  • Current system focused on sweetness and sourness; expanding to other flavor dimensions (e.g., bitterness, umami) is needed.
  • Limited conversational depth in voice-based interaction; future work could integrate affective computing and multimodal feedback.
  • Virtual drink representation lacked synchronization with physical sipping; future systems could incorporate fluid-level tracking.

Summary

XTea is a Mixed Reality system that integrates adaptive virtual environments with physical drinking actions to enrich the bubble tea experience. By leveraging natural language input and sensory cues, XTea enables dynamic personalization, enhances sensory engagement, and fosters social presence. A qualitative study with 12 participants revealed how salient visual changes, voice-based interaction, and narrative cues shaped flavor perception and enjoyment. The paper proposes four design strategies for multisensory beverage experiences, emphasizing dynamic visual cues, user agency, and social dimensions. Future work should address ecological validity, expand flavor dimensions, and refine system responsiveness to enhance immersive beverage experiences.