Using the Visual Language of Comics to Alter Sensations in Augmented Reality

Title of the Paper

Using the Visual Language of Comics to Alter Sensations in Augmented Reality

Paper Information

• Research Domain: Application of augmented reality (AR) and visual language in human-computer interaction
• Keywords: Augmented reality (AR), comics, sensory enhancement, user experience, human-computer interaction, qualitative research, visual symbols, perception manipulation, gamification, user interface

Research Background and Problem Statement

• What issues or challenges did the authors identify?

  • Current augmented reality systems primarily focus on visual perception, making it difficult to directly enhance non-visual senses such as hearing, smell, and touch.
  • There are significant challenges in using existing technologies to enhance the transmission of non-visual sensory information, such as enabling users to feel "stronger" or "faster."

• Why is this problem important?

  • Non-visual senses are equally important in user experience, and enhancing these senses can significantly improve immersion, interactivity, task enjoyment, and efficiency.
  • Addressing this issue can expand the application scenarios of augmented reality technology, such as in education, entertainment, sports, and healthcare.

• Research Motivation and Related Work

  • The authors draw inspiration from the visual language of comics, leveraging the expressive richness of comic symbols to explore their potential in augmented reality.
  • Previous studies have demonstrated the value of comic-style visual symbols in other interactive media (e.g., films, video games), but their role in real-world augmented reality interactions remains underexplored.

Solution

• What methods or solutions did the authors propose?

  • The authors proposed a design space based on comic elements, categorizing comic visual symbols into four sensory categories (auditory, tactile, olfactory/gustatory, proprioception) and three visual expression categories (text, lines, symbols).
  • They developed eight augmented reality prototypes (e.g., using speed lines to convey a sense of "faster," using "bubbles" to make beverages appear more effervescent) and evaluated their effectiveness through user studies.

• What is innovative about this solution?

  • This is the first systematic application of the visual language of comics in augmented reality to enhance non-visual perception.
  • The authors proposed a design framework for designers, establishing connections between sensory categories and visual expressions, pioneering a new direction for integrating comics with augmented reality.

• What are the implementation steps and key technologies used?

  • Design Space Construction: Collecting and categorizing data based on comic research literature to create a matching framework between sensory categories and visual expression categories.
  • Prototype Development: Using Unity engine to implement eight augmented reality application prototypes, utilizing hardware such as Varjo XR-3 and Meta Quest Pro for visual enhancement.
  • User Studies: Recruiting 20 participants and conducting qualitative tests and interviews to evaluate their perception and experience of these augmented reality effects.

Research Findings

• What specific results were achieved?

  • Most participants (at least 75%) correctly understood the meaning of these comic elements and indicated that augmented reality visual effects could alter their perception of non-visual senses.
  • Visual symbols enhanced users' emotional experiences (e.g., more fun, more inspiring), their sense of interaction with the environment, and their engagement in routine tasks.

• What advantages does it have compared to existing solutions?

  • Unlike traditional sensory enhancement technologies (e.g., physical feedback devices), comic symbols can simulate non-visual sensations purely through visual means, offering low-cost and broader applicability.
  • The cartoonish style of visual symbols was widely accepted by users and considered effective in avoiding the "uncanny valley" effect.

• What were the experimental or evaluation results?

  • Auditory enhancement: Musical symbol designs were widely accepted, with 15 participants reporting improved auditory experiences.
  • Tactile enhancement: Vibrating lines and impact explosion effects made 11 participants feel stronger.
  • Proprioception enhancement: Elements like speed lines allowed 13 participants to perceive a "faster" sensation.
  • Gustatory and olfactory enhancement: Seven participants reported that visual symbols made food smell stronger or beverages appear more effervescent.

• Limitations and Future Directions

  • Limitations:
    • The study was limited to laboratory environments, and effects were not tested in everyday life scenarios.
    • The acceptance of visual effects varied among users, with some symbols prone to misinterpretation.
    • Current implementations rely on hardware devices (e.g., tracking cameras), restricting the diversity of practical application scenarios.
  • Future Directions:
    • Conduct quantitative tests to measure the specific impact of these enhancements on user behavior and perception.
    • Investigate the long-term effects of visual symbols on user experience and optimize the matching of symbol effects with object characteristics.
    • Explore personalized approaches to enhance the interpretability and visual appeal of symbols for individual users.
    • Expand research into multi-user environments, such as studying the gamified representation of social interactions enhanced by comic symbols.