GustosonicSense: Towards understanding the design of playful gustosonic eating experiences

Document Title

GustosonicSense: Towards Understanding the Design of Playful Gustosonic Eating Experiences

Document Information

• Subject Area: Human-Computer Interaction (HCI), specifically Human-Food Interaction (HFI) and multisensory experience design
• Keywords: Human-Food Interaction, Gustosonic Experience, Gamification, Eating, Earbuds

Research Background and Problem

• Problems or Challenges:

  1. Existing food-related systems primarily focus on technical implementation or utilitarian goals such as weight loss or reducing food waste, often neglecting the joy and enjoyment of the eating experience itself.
  2. Although previous studies have explored how multisensory experiences, such as sound, can enhance the enjoyment of eating, these studies often require specific sensors and setups, which are invasive or exclusive and lack general applicability.
  3. There is still limited knowledge on how to design gamified multisensory eating experiences.

• Research Importance:

  1. The pleasure and enjoyment of meals are an essential part of life, and understanding and designing gamified eating experiences can enrich daily living.
  2. The HFI field calls for more celebratory and gamified technological designs to enhance multisensory food interactions.

• Research Motivation:

  • Combining the concept of "gustosonic interaction" with celebratory technology to explore the design of a non-invasive, versatile, gamified eating support system.

Solution

• Proposed Method or Solution:

  1. Designed a system called GustosonicSense, which uses wireless earbuds to collect data on users' chewing, drinking, and other oral activities. A machine learning model classifies these activities and plays corresponding playful sounds in real time.
  2. Methodology includes:
     • Using non-invasive wearable earbud devices.
     • Capturing users' temporalis muscle vibrations with built-in six-axis inertial measurement units (IMUs) and microphones.
     • Developing a machine learning model to identify eating activities (e.g., hard foods, soft foods, beverages).
     • Enhancing surprise and interactivity through random audio playback in a mobile application.

• Innovations:

  1. The system supports any type of food or drink without requiring specialized hardware or ingredients.
  2. The system is portable and usable in everyday environments, avoiding invasive or fixed setups.
  3. By designing sounds associated with muscle movements and eating actions, the system stimulates users' playfulness, self-reflection, and multisensory engagement.

• Implementation Steps and Technology:

  1. Data Collection: Gathered data on five oral activities (hard food, soft food, drinking, speaking, and resting) from six participants.
  2. Data Analysis: Trained a random forest model, achieving an F1 score of 0.86 for classification accuracy.
  3. System Architecture: Earbud sensors send IMU data to a mobile application, which predicts sound output via a cloud-based model.
  4. Experimental Design: The application aims to enhance user exploration by playing short sound clips related to eating behaviors, updated every 4 seconds.

Research Outcomes

• Specific Findings:

  1. Demonstrated the functionality of the GustosonicSense system, showing that it can enhance the playfulness of eating through sound.
  2. User Experience Insights:
     • Playfulness (stimulation): Delayed sound playback and mismatched sounds with food characteristics sparked users' curiosity and enjoyment.
     • Hedonism: The system supports self-enjoyment during solo dining, eliminating reliance on visual screens.
     • Reflexivity: Users can freely control the system and create interaction modes based on their needs.
  3. Three Key Design Implications:
     • Employ non-invasive technology to support meaningful eating experiences.
     • Use inconsistency (mismatch between sound effects and food characteristics) as a design resource to spark user interest and exploration.
     • Prioritize users' core values, enhancing personalization and self-expression.

• Comparison with Existing Solutions:

  • The system is more versatile, supporting various food types without relying on specific setups.
  • The non-invasive earbuds are more adaptable for daily use compared to earlier face-mounted or neck-worn sensors.

• Experiment and Evaluation Results:

  • The system achieved an F1 classification accuracy of 0.86, with misclassifications enhancing users' sense of exploration.
  • Users gained insights into their eating rhythms and habit changes through the system's sound feedback and created new interactive eating experiences.

• Limitations and Future Directions:

  • The current participant sample size and demographic diversity are limited; future studies should include broader cultural and social groups.
  • The machine learning model requires further improvement in classification accuracy and has the potential to integrate more food categories.
  • Beyond individual experiences, future research should explore multi-user and social scenarios for eating interactions.
  • Future studies could investigate how to design sound feedback mechanisms that promote healthy eating habits, guiding users toward better dietary practices.

Conclusion

The GustosonicSense system demonstrates the potential of combining earbuds, sound, and machine learning to enhance everyday eating experiences. The findings suggest that adding "gamified" multisensory interactions can not only enrich individual dining experiences but also provide valuable insights for future food interaction technologies and design.