Objestures: Everyday Objects Meet Mid-Air Gestures for Expressive Interaction

Paper Title

Objestures: Everyday Objects Meet Mid-Air Gestures for Expressive Interaction

Publication Info

• Topic area: Integration of everyday object-based and mid-air gesture interactions in mixed reality for expressive and practical applications.
• Keywords: Mixed reality, everyday objects, mid-air gestures, bimanual interaction, interaction design, expressive interaction, user study, design space, tangible interaction, spatial manipulation.

Background and Problem

• Problem / challenge: Existing research on integrating everyday object-based interactions (EOIs) and mid-air gesture interactions (MAIs) often focuses on narrow use cases or specific technologies, lacking generalizable guidance for designers and developers.
• Significance: The integration of EOIs and MAIs can enhance mixed reality experiences by leveraging complementary affordances, such as tactile feedback from objects and the convenience of mid-air gestures.
• Motivation and related work: Prior work has explored EOIs and MAIs separately or in limited combinations, often constrained by specific objects or technologies. This paper aims to formalize a design space that unifies these modalities for broader applicability.

Solution

• Proposed approach: Objestures—a framework of five interaction types (Binary, Linear, Rotational, Nonlinear, Free) that span EOIs and MAIs, enabling expressive uni- and bimanual interactions.
• Novelty:
  1. Formalization of a design space combining EOIs and MAIs with five interaction types.
  2. Implementation and evaluation of Objestures through a prototype system.
  3. Exploratory user study demonstrating performance comparable to native freehand manipulation.
  4. Case studies showcasing expressive applications across diverse contexts.
• Procedure and key techniques:
  1. Define five interaction types based on affordances and kinematic constraints.
  2. Map EOIs to corresponding MAIs using pantomimic gestures.
  3. Evaluate the framework through a user study on 3D tasks (scaling and rotation) and case studies in applications like music, drawing, and shadow exploration.
  4. Prototype interactions using hand tracking and object manipulation in mixed reality.

Results

• Concrete findings:
  • Objestures achieved performance on scaling and rotation tasks comparable to native freehand manipulation, with lower error and movement.
  • Participants rated Objestures as intuitive, engaging, and expressive, with interest in everyday use.
  • Median preference rankings favored Objestures with physical objects (Obj) over purely virtual interactions (NObj) and native freehand manipulation (Hands).
• Advantage over baselines:
  • Obj and NObj significantly reduced error and movement compared to Hands.
  • Obj provided better accuracy and stability due to physical anchoring.
• Experiments / evaluation:
  • User study (N = 12) on scaling and rotation tasks with three approaches (Hands, Obj, NObj) across near and far distances.
  • Case studies on three applications (Sound, Draw, Shadow) covering all five interaction types.
  • Metrics included error, movement, time, and subjective user experience (NASA-TLX, UEQ-S).
• Limitations and future work:
  • Small sample size (N = 12) limits generalizability.
  • Physical object constraints (e.g., range of motion, stability) and tracking limitations (e.g., occlusion, jitter) need improvement.
  • Future work could explore larger-scale evaluations, additional interaction types, and extensions to other body parts or modalities.

Summary

Objestures integrates everyday object-based and mid-air gesture interactions into a unified framework of five interaction types, enabling expressive and practical mixed reality applications. Through a user study and case studies, the framework demonstrated comparable performance to native freehand manipulation and was rated positively for intuitiveness and engagement. Objestures provides designers with a structured design space to explore and combine EOIs and MAIs, addressing gaps in prior work. Future research could expand the framework to include more interaction types, larger user studies, and additional modalities.