InkFlow: Connected Handwriting Recognition for Natural Mid-Air Input in Mixed Reality

Paper Title

InkFlow: Connected Handwriting Recognition for Natural Mid-Air Input in Mixed Reality

Publication Info

• Topic area: Mid-air handwriting recognition in Mixed Reality (MR) environments.
• Keywords: Mixed Reality, mid-air handwriting, stroke recognition, DS-TCN, meta-learning, kinematic features, user adaptation, natural interaction, real-time processing, usability evaluation.

Background and Problem

• Problem / challenge: Existing mid-air handwriting systems rely on explicit control mechanisms (e.g., gestures, buttons, or planar proxies) that disrupt natural writing flow, impose cognitive load, and cause fatigue. These systems often lack robust generalization across users and require specialized hardware.
• Significance: Natural handwriting is essential in MR for applications like immersive teaching, collaborative whiteboarding, and digital signatures, as it preserves spatial and stylistic characteristics. Improving the naturalness and fluency of mid-air handwriting can enhance user experience and broaden MR applications.
• Motivation and related work: Prior methods focus on symbolic text input or single-character handwriting, often requiring manual control or specialized devices. These approaches fail to support continuous, natural handwriting in 3D space. InkFlow addresses this gap by enabling automatic stroke recognition from continuous hand movements without explicit control mechanisms.

Solution

• Proposed approach: InkFlow, a kinematics-driven handwriting recognition system for MR, enables natural and continuous mid-air handwriting without explicit gesture switching or specialized hardware.
• Novelty:
  1. A user-friendly data collection pipeline using Pinch-Release gestures for self-annotated handwriting data.
  2. A lightweight dual-head DS-TCN model with boundary-aware alignment for robust stroke recognition.
  3. A meta-learning framework with domain generalization for cross-user applicability and rapid adaptation with minimal data.
  4. Real-time asynchronous visual feedback for seamless handwriting interaction.
• Procedure and key techniques:
  1. Data collection via Pinch-Release gestures to label pen-up/pen-down states.
  2. Training a DS-TCN model using kinematic features (velocity, acceleration) and soft boundary labels.
  3. Meta-learning with cross-domain generalization to optimize for user-specific handwriting styles.
  4. Real-time stroke recognition with sliding-window processing and asynchronous feedback to mitigate latency.

Results

• Concrete findings:
  • Achieved frame-level accuracy of 84.16% (Accraw) and 88.36% (Acctol), and segment-level F1@0.5 of 78.54%.
  • InkFlow improved handwriting speed by 22.9% for English and 51.6% for Chinese compared to Pinch-Release.
  • Reduced physical demand and effort in NASA-TLX scores compared to baseline methods.
• Advantage over baselines:
  • Outperformed Virtual-Plane and Pinch-Release methods in usability, efficiency, and user preference.
  • Demonstrated strong cross-user generalization and rapid adaptation with few-shot personalization.
• Experiments / evaluation:
  • Comparative user study (N=30) evaluated usability and preference against baseline methods.
  • Closed-loop online study (N=12) assessed real-world performance with error correction.
  • Ablation studies validated the effectiveness of boundary-aware design and meta-learning.
• Limitations and future work:
  • Fixed sliding-window context limits recognition of extremely large, slow, or fast handwriting.
  • Requires users to face written content for correct coordinate transformation.
  • Limited evaluation across diverse languages and writing systems.
  • Challenges in optimizing visual accessibility from different spatial angles.

Summary

InkFlow is a novel system for natural mid-air handwriting in Mixed Reality, leveraging kinematic features, a DS-TCN model, and meta-learning for robust stroke recognition. It eliminates explicit gesture control, enabling continuous handwriting with reduced fatigue and improved efficiency. User studies demonstrated significant advantages over traditional methods in usability and comfort. While limitations exist in handling diverse writing behaviors and languages, InkFlow shows promise for applications in education, collaboration, and creative design.