Exploring Mobile Devices as Haptic Interfaces for Mixed Reality

Title of the Paper

Exploring Mobile Devices as Haptic Interfaces for Mixed Reality

Paper Information

• Research Domain: Mixed Reality Interaction Technology, Haptic Interface Design, and User Experience
• Keywords: Mobile gestures, haptic feedback, mobile devices, mixed reality, gesture guidance, haptic exploration, haptic interfaces

Research Background and Problem

• Problems and Challenges:
  • Existing mixed reality (MR) systems often rely on handheld controllers to provide haptic experiences. However, as mobile MR becomes more prevalent, using dedicated controllers in public spaces becomes inconvenient.
  • While hand tracking capabilities of head-mounted displays (HMDs) reduce hardware dependencies, the lack of tangible input devices limits haptic exploration and immersion.
• Significance:
  • Mobile MR is transforming traditional deployment scenarios from home or dedicated spaces to public environments. Haptic interaction in public spaces is particularly critical, such as for mobile collaboration, urban planning, or autonomous vehicles.
  • Transforming mobile devices into MR controllers can enhance technological accessibility and flexibility while offering rich haptic experiences.
• Related Work:
  • Previous studies have explored MR interactions using everyday objects, smartwatches, and touch-based devices. However, there is a lack of in-depth research on users' gesture requirements and actual haptic feedback needs when exploring virtual objects.

Solution

• Method or Solution:
  • Proposes using conventional mobile devices (e.g., smartphones) as MR controllers, combined with haptic feedback technology, to explore their potential in virtual object interaction.
  • Conducts two user studies, including an online survey (N=50) and a gesture-guided study (N=18), to analyze how users perceive haptic attributes of virtual objects through mobile devices.
• Innovations:
  • Identifies three core modes of mobile devices in haptic exploration: devices as substitutes for virtual objects (object mode), as substitutes for hands (hand mode), and as tools (tool mode).
  • Provides a comprehensive user-defined gesture set and haptic feedback design space, encompassing over 200 gestures and 11 types of haptic feedback.
• Implementation Steps and Techniques:
  1. Conduct an online survey to evaluate people's expectations of haptic attributes of virtual objects based on visual information.
  2. Perform a gesture-guided study to delve into users' natural gestures and device-based haptic exploration of virtual objects.
  3. Summarize key gesture types and haptic feedback, constructing a large-scale interaction design space.

Research Results

• Specific Results:
  • Develops 10 primary gesture types, including "grasp and move," "two-handed grasp," "pinching," "tapping," and others.
  • Designs 11 types of haptic feedback, including force feedback, surface deformation, weight sensing, temperature radiation, and more.
  • Defines three modes for mobile devices in MR: object mode, hand mode, and tool mode.
• Advantages Comparison:
  • Transforms everyday devices into haptic interfaces, enhancing hardware flexibility and accessibility.
  • Compared to vibration feedback controllers dedicated to MR, this solution provides broader and more precise haptic effects.
• Experimental and Evaluation Results:
  • Experiments show that user-preferred gesture designs and haptic feedback significantly enhance the experience of exploring objects in MR.
  • Users tend to combine multiple haptic attributes (e.g., temperature and weight) for a more comprehensive virtual object experience.
• Limitations and Future Directions:
  • The current study only explores the potential of smartphones as haptic devices, without addressing other device forms (e.g., smartwatches or tablets).
  • The focus of this research is on the exploration of single haptic attributes, which could be expanded to multi-attribute joint analysis in the future.
  • Social acceptability in public spaces and differences between left-handed and right-handed users require further validation.
  • Future development of smartphones with multi-form transformation mechanisms and real-time haptic feedback is recommended to support more precise MR interaction experiences.

This study provides a critical foundation for constructing haptic interaction interfaces for mobile devices in mixed reality and offers theoretical and practical support for designing new mobile MR devices and interaction modes.