HOICraft: In-Situ VLM-based Authoring Tool for Part-Level Hand-Object Interaction Design in VR

Paper Title

HOICraft: In-Situ VLM-based Authoring Tool for Part-Level Hand-Object Interaction Design in VR

Publication Info

• Topic area: AI-assisted authoring tools for VR hand-object interaction design.
• Keywords: Hand-object interaction, virtual reality, in-situ authoring, vision-language models, interaction design, customization, user experience, AI-assisted tools, VR prototyping, part-level interaction.

Background and Problem

• Problem / challenge: Designing part-level hand-object interactions (HOI) in VR is labor-intensive, requiring manual trial-and-error to align with user abilities and design intent. Existing tools lack intelligent support for part-level interaction mapping and customization.
• Significance: Effective HOI design is crucial for realistic and engaging VR experiences, but current workflows are time-consuming and inconsistent, limiting productivity and user satisfaction.
• Motivation and related work: Previous research has explored object motion mechanics, gesture-based interaction, and AI-assisted authoring, but these approaches fail to address fine-grained part-level HOI design. Intelligent tools leveraging user data and foundational models are needed to streamline this process.

Solution

• Proposed approach: HOICraft, an AI-assisted in-situ authoring tool that uses vision-language models (VLMs) and user preference data to support part-level HOI design in VR.
• Novelty:
  1. Introduction of a VLM-based system for interactive part selection and HOI mapping.
  2. Identification of five representative HOI design criteria derived from formative studies.
  3. Development of a user-preference-driven HOI recommendation module using in-context learning.
  4. Integration of customization features for fine-tuning HOI properties in an in-situ VR environment.
• Procedure and key techniques:
  • Analyze 3D objects using VLMs to identify interactive parts and affordances.
  • Provide ranked recommendations for HOI designs based on user preference data and design intent.
  • Allow designers to customize interaction parameters (e.g., resistance, gestures, animation) and test them in real-time.
  • Use a hybrid decision-making approach (ranking-based and binary) to map interaction designs to user experience goals.

Results

• Concrete findings:
  • HOICraft reduced trial-and-error iterations compared to manual workflows.
  • Designers achieved outcomes comparable in quality to manual methods with less effort.
  • System usability was rated highly (SUS score: 81.04 ± 13.75).
• Advantage over baselines:
  • Reduced exploratory counts (trial-and-error) significantly (p < 0.001).
  • Comparable decision speed to manual methods but with lower cognitive and physical burden.
  • Recommendations aligned well with user intent and supported diverse design goals.
• Experiments / evaluation:
  • Formative study with 3 VR developers identified key challenges and HOI patterns.
  • User study with 20 participants collected empirical data on HOI preferences and performance.
  • Evaluation with 12 VR designers compared HOICraft to manual workflows and assessed usability.
• Limitations and future work:
  • Current focus on simple prismatic and revolute joints; future work will address complex kinematics and multi-part coordination.
  • Reliance on preprocessed 3D objects; plans for hybrid or fully automated preprocessing pipelines.
  • Extension to broader XR environments and diverse runtime platforms is needed.

Summary

HOICraft is an AI-assisted in-situ authoring tool designed to streamline part-level hand-object interaction (HOI) design in VR. By leveraging vision-language models and user preference data, it provides intelligent recommendations, customization options, and real-time testing capabilities. Empirical studies demonstrated that HOICraft reduces trial-and-error iterations, supports diverse user intents, and achieves outcomes comparable to manual workflows with less effort. While currently focused on simple interactions, future work aims to expand its applicability to complex objects, automated preprocessing, and broader XR contexts.