SandTouch: Empowering Virtual Sand Art in VR with AI Guidance and Emotional Relief

Research Background and Problem Statement

• Identified Problems or Challenges:

  • Sand painting is a unique art form, but its popularity is limited due to the need for expensive equipment, significant material consumption, difficulty in preservation, and spatial constraints.
  • Sand painting involves complex, personalized manual techniques, making it challenging for beginners to learn and participate.
  • Existing digital sand painting solutions rely on specialized hardware or simple gesture controls, lacking refined tools, auditory feedback, and immersive interaction, which restrict users' creative freedom and experience.

• Importance of the Problem:

  • As an art form capable of expressing intense emotions, sand painting holds significant aesthetic value, but its technical and equipment barriers hinder its promotion and education.
  • Providing a convenient, innovative digital sand painting solution could attract more users to participate, reduce learning costs, and enhance creativity and emotional expression.

• Research Motivation and Related Work:

  • By integrating virtual reality (VR), artificial intelligence (AI), and gesture recognition technologies, the study aims to provide a more natural, diverse, and immersive environment for sand painting creation.
  • The authors referenced existing VR art tools and sand painting simulation systems, such as Google Tilt Brush and SandCanvas, identifying their limitations and proposing a more flexible and user-friendly interaction approach.

Solution

• Proposed Solution:

  • SandTouch: A VR-based sand painting system that simulates the interactive experience of real sand painting through gesture recognition technology, combined with auditory feedback and AI guidance to enhance user creativity and emotional expression.

• Innovative Aspects of the Solution:

  • Developed a natural gesture library to accurately simulate real sand painting actions, such as "dotting," "pouring," and "hand wiping," enhancing the naturalness and precision of interactions.
  • Implemented real-time rendering of sand dynamics and introduced sand pen tools with droplet and scatter effects to improve the detail of user creations.
  • Integrated a large language model (LLM) to provide creative guidance, tutorials, and intelligent evaluation tools to help users improve their artistic skills.
  • The system supports emotional regulation through detailed auditory feedback and an immersive environment to alleviate stress.

• Implementation Steps and Key Technologies:

  • Gesture Recognition: Utilized PICO VR devices to capture 26 hand keypoints, combined with the lightweight MobileNet V3 neural network to recognize hand movements, optimizing user interaction.
  • Sand Dynamics Simulation: Designed a height-field-based sand dynamics simulation algorithm, leveraging GPU acceleration for collision detection, sand dispersion, and erosion.
  • Auditory Feedback: Developed an auditory feedback system in Unity, dynamically adjusting the volume and pitch of sand sounds based on user gestures to enhance immersion.
  • AI Support: Leveraged the GPT4o model to provide intelligent creative guidance, including element decomposition, painting structure analysis, and real-time suggestions.

Research Outcomes

• Key Achievements:

  • User experiments with SandTouch demonstrated its exceptional performance in interaction flexibility, creative expression, immersion, and emotional release.
  • The system significantly improved user engagement and satisfaction in immersive sand painting, precise gesture recognition, and flexible creative guidance.
  • Delivered a highly accurate gesture recognition system (99.8%) with robust performance under various environmental conditions.

• Advantages Over Existing Solutions:

  • More natural and precise sand painting gesture interaction design, surpassing touch-based or hardware-dependent sand painting solutions.
  • Deep integration of AI and VR effectively reduces the learning curve for beginners while supporting advanced creators in exploring intelligent creative possibilities.
  • Auditory feedback significantly enhances immersion and emotional regulation, an area often overlooked by many existing tools.

• Experimental or Evaluation Results:

  • In user trials, the SandTouch system received high scores for interaction, creativity, and emotional release, significantly reducing the learning curve for beginners.
  • In stress and emotional regulation tests, participants experienced a 35% reduction in stress levels and a 38.3% decrease in depression levels after one week of use.

• Limitations and Future Directions:

  • Limitations:
    • Hand interactions are occasionally not smooth, and response delays may affect detailed drawing.
    • The system lacks diverse background scenes and customizable gesture functions.
    • The tutorial design is not sufficiently detailed and could benefit from dynamic gesture guidance.
  • Future Directions:
    • Integrate haptic feedback (e.g., vibrating gloves) to enhance the realism of sand painting.
    • Expand painting tools and customizable gesture features to support personalized creative needs.
    • Conduct long-term evaluations of SandTouch's emotional therapeutic effects and incorporate gamification to improve user engagement.
    • Optimize existing algorithms to further reduce interaction response delays and improve the quality of sand dynamics rendering.

Conclusion

SandTouch leverages AI and VR to provide users with an immersive sand painting experience while helping beginners quickly learn and express emotions. This research demonstrates how technology can lower the barriers to traditional art forms and opens new possibilities for digital art and emotional regulation.