TouchPose: Hand Pose Prediction, Depth Estimation, and Touch Classification from Capacitive Images

Document Title

TouchPose: Hand Pose Prediction, Depth Estimation, and Touch Classification from Capacitive Images

Document Information

• Subject Area: Human-Computer Interaction, Gesture Recognition, and Touch Technology
• Keywords: Hand Pose Prediction, Depth Estimation, Capacitive Imaging, Multi-Touch, Human-Computer Interaction, Finger Classification

Research Background and Problem

• Identified Problem/Challenge: Current touchscreen devices can only detect the 2D coordinates of user touch inputs, neglecting the complex 3D configuration of the hand, which limits the ability to fully capture user gestures.
• Significance of the Problem: Efficient reconstruction of 3D hand skeletons can introduce richer interaction methods for touch devices, with applications in augmented and virtual reality, rehabilitation therapy, and human-computer interaction design.
• Research Motivation: Existing studies primarily rely on optical sensors or external devices (e.g., gloves and depth cameras) to reconstruct hand poses, lacking solutions for directly predicting 3D gestures using everyday touchscreens.
• Related Work:
  • Traditional methods use RGB and depth images for hand pose estimation.
  • Research on touch imaging technology has been limited to 2D touch shapes or simple finger angles.
  • Gesture recognition applications face challenges such as data sparsity and distinguishing between touch and hovering fingers.

Solution

• Proposed Method: TouchPose, a deep learning-driven model that uses capacitive images to predict 3D hand poses and depth maps while classifying touch events as fingertip or whole-hand touches.
• Innovative Aspects of the Solution:
  • Introduced the first 3D hand skeleton prediction model based on capacitive images.
  • A multi-task learning framework capable of simultaneous hand pose prediction, depth estimation, and touch classification.
  • Directly infers hand configurations from capacitive imaging without requiring additional hardware support.
• Implementation Steps:
  • Data Collection: Built a dataset of 65,374 sample pairs containing capacitive images, corresponding 3D hand position annotations, and depth maps.
  • Model Construction: Utilized a U-Net-shaped convolutional neural network architecture with a shared embedding space for multi-task learning.
  • Model Training: Employed a gradient descent optimizer, processed in batches, and conducted multiple training iterations.
  • Validation and Testing: Validated model performance across cross-user, cross-gesture, and cross-session scenarios.

Research Outcomes

• Specific Results:
  • Reconstructed 3D hand skeletons with an average endpoint error of 21.8 mm.
  • Achieved an average depth map error of 22.2 mm.
  • Predicted 3D joints outside the touch area.
• Advantages:
  • Does not require external sensors or additional devices compared to existing methods.
  • Demonstrates strong generalization across different users, gestures, and unseen samples.
• Experimental Results:
  • Finger classification accuracy reached up to 91.1%, slightly dropping to 83.1% in cross-gesture testing.
  • Finger angle prediction errors averaged 10.4° (yaw) and 9.6° (pitch).
  • Enhanced touchpad recognition resolution and provided 3D gesture data.
• Limitations and Future Directions:
  • The dataset is limited to the right hand and does not support multi-hand interactions.
  • Ambiguities exist in single-touch finger classification.
  • Higher touchscreen resolution and broader sensing range are needed to improve reconstruction accuracy.
  • Expanding to more devices and dynamic gesture scenarios requires additional data recording and training.
  • Exploring generative methods to provide multiple possible hand pose solutions.

Additional Analysis

• Application Scenarios:
  • Enables gesture-based interaction operations in user interface design.
  • Supports 3D control in augmented reality and virtual reality.
  • Prevents accidental touches and enhances user experience on large-screen devices.
• Open Resources: The dataset and model have been made publicly available to support future researchers in reproducing and extending the work.

This document provides significant insights into innovative applications of touch technology in human-computer interaction scenarios and demonstrates how sparse sensor data can be effectively utilized for 3D pose reconstruction.