ProGesAR: Mobile AR Prototyping for Proxemic and Gestural Interactions with Real-world IoT Enhanced Spaces

Title of the Paper

ProGesAR: Mobile AR Prototyping for Proxemic and Gestural Interactions with Real-world IoT Enhanced Spaces

Paper Information

• Research Domain: Human-Computer Interaction; Prototyping for proxemic and gestural interactions in IoT-enhanced spaces using mobile augmented reality technology
• Keywords: Proxemic interaction, Gestural interaction, AR prototyping, Mobile augmented reality, User interface design, Spatial interaction design

Research Background and Problems

• Identified Problems or Challenges

  • Current IoT-enhanced spaces require designing proxemic and gestural interactions between users and objects, but commonly used prototyping tools have limitations:
    1. Many tools rely on specialized hardware (e.g., motion capture devices) or require programming skills;
    2. Current augmented reality tools primarily focus on first-person experience prototyping rather than whole-body interaction perspectives;
    3. Few tools support testing from both first-person and third-person perspectives.
  • Designers find it difficult to transition from 2D prototypes to imagining interactions in 3D real-world scenarios, lacking design tools with physical realism.
  • There is a lack of low-cost, highly interactive, and easily repeatable prototyping tools.

• Research Significance

  • Interaction design in IoT spaces not only impacts user experience but also requires careful consideration of the rationality of interactions with physical spaces.
  • An efficient mobile platform design tool can reduce the time cost of design and testing, improving the performance of interaction design in real-world spaces.

• Research Motivation and Related Work

  • Current AR prototyping tools are mostly focused on visual presentation, emphasizing the overlay of dynamic digital content in user experiences, but provide limited support for whole-body interaction in physical environments.
  • Existing research on proxemic and gestural interaction design (e.g., Proximity Toolkit) faces challenges such as hardware dependency and high software programming barriers.
  • There is a need to develop a low-threshold, flexible, and mobile-compatible prototyping tool to inspire further exploration by other researchers.

Proposed Solution

• Proposed Solution

  • Developed ProGesAR, a mobile AR design tool that supports both third-person and first-person testing perspectives, enabling the creation and testing of proxemic and gestural interaction prototypes in real IoT environments.
  • ProGesAR supports the triggering of dynamic content and allows designers to quickly build low-cost interaction prototypes using an "event-effect" mapping method.
  • Provides a declarative interface that requires no programming skills to use.

• Innovations

  • Offers in-situ interaction design support in real-world scenarios, making designs more aligned with actual usage environments.
  • Supports multiple event-triggering mechanisms (position, orientation, distance, and gesture events), enhancing the expressiveness of interactions.
  • Enables multi-perspective testing (user first-person perspective and observer third-person perspective), covering the entire process from design to concept validation.

• Implementation Steps and Key Technologies

  1. Authoring Mode: Designers place virtual assets in the scene using a mobile device and orchestrate events and dynamic effects;
  2. Event Detection: Supports detection of position, orientation, distance, and body posture;
  3. Testing Mode: Allows designers to test prototypes themselves or invite participants to observe experimental effects from different perspectives;
  4. Underlying Technology: Built on iOS devices and ARKit, enabling real-time motion tracking, plane detection, and basic gesture classification.

Research Outcomes

• Specific Outcomes

  • Successfully developed the ProGesAR tool and demonstrated its capability for rapidly creating and testing low-fidelity prototypes in near-realistic scenarios;
  • In user studies, participants quickly designed rich interaction scenarios and applied them to various IoT contexts (e.g., smart homes, public facilities, game design).

• Comparison with Existing Solutions

  • Compared to traditional tools, ProGesAR's low-cost threshold and mobility make it more suitable for early-stage rapid iterative design;
  • Supports multi-perspective testing from both first-person and third-person views, addressing the lack of multi-perspective support in traditional interaction tools.

• Experimental or Evaluation Results

  • Users were able to design complete interaction prototypes in a short time (average of 157.5 seconds per scenario).
  • Users expressed satisfaction with ProGesAR through System Usability Scale (SUS) scores, particularly praising its multi-perspective testing, real-time interaction, and low learning curve.

• Limitations and Future Directions

  • The current system primarily supports single-user interactions; future work could expand to multi-user scenarios.
  • The range of gestures is relatively limited; subsequent work could include a more extensive and customizable gesture library.
  • The limited screen size of mobile devices may affect the coherence of interactions in complex scenarios; future improvements should focus on optimizing the visual interface and supporting remote design.