ProInterAR: A Visual Programming Platform for Creating Immersive AR Interactions

Title of the Paper

ProInterAR: A Visual Programming Platform for Creating Immersive AR Interactions

Paper Information

• Subject Area: Augmented Reality (AR) application development, visual programming, user interface design
• Keywords: AR content, AR interaction, visual programming, augmented reality, user interface, tablet and AR-HMD integration, AR application prototyping

Research Background and Problem Statement

• Problems and Challenges: Developing AR applications requires designing diverse interactions between real and virtual content, which demands high programming skills. Creators with limited coding experience face difficulties in scripting, content migration, and testing/debugging processes.
• Significance: AR technology is being widely applied in entertainment, education, healthcare, and industry. Lowering development barriers can reduce technical thresholds, enabling beginners to easily design AR applications with complex interactions.
• Related Work: Existing AR tools are often limited to specific domains (e.g., gesture interaction, toy binding) or only support interactions with virtual content, failing to meet the programming needs for complex behavior logic between real and virtual content. Additionally, many tools only support desktop programming and virtual scene creation, creating a disconnect between development and actual testing.

Proposed Solution

• Approach: A visual programming platform called ProInterAR is proposed, combining tablets and AR-HMD devices to provide an intuitive interaction programming environment for novice AR developers.
• Innovations:
  • Supports rapid definition of AR content interaction behaviors through a block-based programming interface on tablets.
  • Provides direct support for 3D spatial interactions between real and virtual content.
  • Offers a portable and integrated development mode: creating content from the AR-HMD perspective, programming on a tablet, and executing and testing in real-time within AR scenes.
  • Enables rich interactions and conditional logic through modular programming components, such as conditional statements, nested loops, and variable settings.
• Implementation Steps and Core Technologies:
  1. Scene Creation Interface:
     • Add real objects, virtual content, and environments to AR scenes using HoloLens 2.
     • Provides convenient content creation features, such as gesture recognition, plane detection, and 3D mesh selection.
  2. Visual Programming Interface:
     • Runs on a tablet browser, designed based on Scratch. Main modules include Motion, Looks, Control, Sensing, Variables, and other nine categories.
     • Supports drag-and-stack operations and establishes independent or synchronized interaction behaviors through event-trigger mechanisms.
  3. Execution and Control Interface:
     • Used to run programs and observe and operate in real-time within AR scenes. Supports dynamic adjustments by freely switching between AR-HMD and tablet.
  4. Communication Architecture:
     • Uses WebSocket for real-time communication between the tablet and HoloLens, with a server facilitating message transmission and data synchronization.

Research Outcomes

• Specific Achievements:
  • Provides an efficient programming tool for quickly designing and validating AR applications, covering various use cases such as AR games, teaching, animation, and information visualization.
  • Demonstrated application scenarios in experiments, such as "3D Whack-a-Mole," virtual sound instruments, interactive physics lessons, and multi-step animations.
• Advantages Analysis:
  • Compared to Traditional Tools: Achieves an immersive programming experience with instant feedback, avoiding tedious content migration and switching between desktop and AR devices.
  • Compared to Existing AR Tools: Supports a wider range of interaction types, including complex logic and diverse scenarios between real and virtual content.
• Experiments and Evaluation:
  • User Study 1: 12 participants created 22 AR applications covering entertainment and education. Results showed that ProInterAR has a low learning curve for beginners, with high scores in usability (e.g., SUS survey) and user satisfaction.
  • User Study 2 (Long-term Evaluation): Two users with different backgrounds (no programming experience and experienced developers) completed an office AR assistant and an adventure game project over five days. The study showed significant improvement in user proficiency with continuous use.
• Limitations and Future Directions:
  • Limitations:
    1. Debugging and cross-content communication become cumbersome as interaction logic complexity increases.
    2. Lacks support for physical simulations (e.g., gravity) and rich environmental context information (e.g., time, weather).
    3. Limited tracking capabilities for automatically moving physical objects.
  • Improvements:
    • Add debugging tools and more intuitive logic node programming features.
    • Provide predefined composite function modules (e.g., virtual joysticks).
    • Integrate more flexible physical simulation and scene detection algorithms.
• Potential Applications:
  • Promote in education for computational thinking and creative programming teaching.
  • Extend to industrial use cases, such as IoT functionality design and interactive prototype development.

Conclusion: ProInterAR demonstrates an efficient design for an augmented reality interaction programming platform. Its usability and flexibility validate its potential applications across diverse fields, providing a novel tool for promoting low-barrier AR application development.