InfinitePaint: Painting in Virtual Reality with Passive Haptics Using Wet Brushes and a Physical Proxy Canvas

Document Title

InfinitePaint: Painting in Virtual Reality with Passive Haptics Using Wet Brushes and a Physical Proxy Canvas

Document Information

• Subject Area: Human-Computer Interaction and Digital Painting Systems in Virtual Reality
• Keywords: Digital Painting, Virtual Reality, Mixed Reality, Passive Haptic Feedback, Traditional Art, Brush Input

Research Background and Problem

• Identified Issues or Challenges:

  • Traditional digital drawing tablets can simulate parameters like brush pressure but fail to provide the tactile experience of using a real wet brush on paper.
  • Although current simulation environments have improved in graphics and visual effects, they still lack tools that preserve the tactile sensations of real-world painting.
  • Virtual and digital painting often overlook the subtle tactile interactions inherent in artistic mediums like watercolor.

• Research Significance:

  • Address the gap in providing traditional artists with a realistic tactile experience on digital platforms.
  • Offer a cost-effective and infinite painting method while integrating digital functionalities (e.g., undo, special effects).

• Motivation and Related Work:

  • Existing research focuses on brush simulation and haptic feedback but rarely incorporates the physical properties of real wet brushes into VR systems.
  • Researchers have attempted to combine watercolor techniques with digital tracking, but most solutions rely on expensive professional hardware.

Solution

• Summary of Method or Solution:

  • Propose the InfinitePaint system, which integrates wet brushes, special paper, and camera capture technology to enable digital painting in virtual reality environments.
  • Users paint on special paper that turns black when wet and reverts to its original color upon drying.
  • Cameras capture real-time brush strokes, digitizing them and rendering them in the VR environment with selected colors.

• Innovations:

  • Utilize special wet paper ("magic paper") that requires no additional ink or painting materials.
  • Real-time capture and virtualization of the tactile and visual effects of wet brushes.
  • Support infinite painting through the drying cycle of the paper.
  • Low-cost system design that does not rely on expensive hardware.

• Key Implementation Steps and Techniques:

  1. Hardware Configuration:
     • Use Azure Kinect (camera) to capture user brush strokes and generate virtual views via Unity.
     • Employ low-cost wet paper that turns black when wet as the physical tactile canvas.
     • Users interact with the virtual environment through VR headsets (e.g., HTC Vive).
  2. Real-Time Processing:
     • Apply perspective correction (homography) to transform camera input into canvas projections.
     • Capture real-time brush strokes and extract user-generated images using background subtraction.
     • Combine HSV space with a color selector to allow users to choose digital effects in real time.
  3. Drawing Features:
     • Implement basic brush functions (drawing, color selection) and advanced features (blurring, layer manipulation).
     • Capture special operations like "water dripping" and "ink blowing" to support complex artistic expressions.

Research Outcomes

• Main Results:

  • Developed the InfinitePaint prototype system, enabling users to experience digital painting with wet brushes in VR.
  • Conducted user studies demonstrating the system's superiority in inspiration and artistic expression compared to traditional tablet drawing.

• Advantages:

  • Compared to traditional digital tools, InfinitePaint offers a more natural tactile experience.
  • The disappearing wet brush canvas avoids the waste associated with traditional canvases.
  • Users can experience traditional artistic features like the "unexpectedness of watercolor" in digital painting.

• Experimental or Evaluation Results:

  1. User Experience:
     • The experiment involved 14 artists (including amateurs and professionals) in quantitative and qualitative studies.
     • Users rated InfinitePaint significantly higher in "stimulation" and "novelty" compared to traditional drawing tablets.
     • Most users felt its tactile experience closely resembled that of real wet brushes and expressed willingness to use it in the future.
  2. Specific Painting Effect Demonstrations:
     • Artwork produced during the experiment showcased how InfinitePaint captures details like dry brush textures and feathered edges.
     • Free painting sessions resulted in several highly expressive artistic creations.

• Limitations and Future Directions:

  • Current Limitations:
    • The virtual canvas's positional drift feels slightly unnatural, affecting immersion.
    • The system does not address wet-on-wet techniques, limiting support for certain watercolor characteristics.
    • The resolution and weight of VR headsets may impact user comfort.
  • Future Improvements:
    • Optimize the virtual canvas "drift" mechanism, such as using haptic repositioning techniques.
    • Integrate real-world physics with digital simulation to support more traditional artistic techniques.
    • Enhance user experience for advanced features like color blending tools and 3D object texture projection.
    • Explore possibilities for non-VR versions and portable hardware.

Conclusion

InfinitePaint combines the tactile experience of traditional painting with the advantages of digital technology, enabling infinite painting and highly expressive artistic creation in VR environments. By optimizing user experience and expanding functionality, it has the potential to be widely adopted by both professional and amateur artists.