Palmpad: Enabling Real-Time Index-to-Palm Touch Interaction with a Single RGB Camera

Research Background and Problem Statement

• Identified Problems and Challenges:

  1. In mixed reality (MR) environments, interacting with the opponent's palm (e.g., tapping the opponent's palm with the index finger) is a convenient interaction method. However, existing vision-based gesture tracking systems, especially those using only a single RGB camera, perform poorly in accurately recognizing touch states.
  2. Current MR devices (e.g., Microsoft HoloLens 2 and Meta Quest 2), although equipped with depth cameras or multi-camera setups, lack sufficient accuracy in depth estimation between fingers and palms, making it difficult to distinguish between an index finger touching the palm and hovering slightly above it.
  3. Some studies have addressed this issue by using additional wearable devices or sensors, but excessive hardware may increase user discomfort.
  4. The action of tapping the palm with a finger not only requires recognizing whether a touch has occurred but also demands sensing dynamic information such as the force and speed of the tap, which existing methods struggle to capture precisely.

• Significance:

  1. Accurate finger-tap recognition is crucial for touch interactions in MR devices, such as virtual keyboard input or swipe operations.
  2. Single-camera solutions reduce hardware complexity and cost, making them more suitable for lightweight MR devices.
  3. Touch interactions provide an efficient and readily available virtual input interface, enhancing user experience.

• Research Motivation and Related Work:

  1. Current solutions (e.g., depth camera-based or multi-camera methods) provide relatively high spatial resolution but are limited by the complexity of the hardware.
  2. Some approaches combine time-series models (e.g., LSTM) or pressure-sensing technologies to achieve touch recognition, but their performance has not met expectations.
  3. The authors propose Palmpad, which addresses the limitations of existing vision technologies in lightweight MR devices by providing efficient, real-time touch detection using a single RGB camera.

Solution

• Proposed Method and Innovations:

  1. A novel method for real-time detection of index finger tapping on the palm using a single RGB camera, named Palmpad.
  2. Motion information between frames is extracted using optical flow technology, significantly improving the accuracy of touch state determination.
  3. A convolutional neural network (CNN) is used for touch state classification, with optical flow, palm local features, and finger local features as inputs.

• Innovations:

  1. The first application of optical flow technology for intra-frame palm touch recognition.
  2. Precise localization of palm and finger regions using MediaPipe, reducing computational load and redundant processing.
  3. Achieves superior touch classification accuracy (97.0%) compared to existing methods.
  4. The method requires only a single RGB camera, offering a more lightweight hardware configuration compared to traditional depth-based solutions.

• Implementation Steps and Key Techniques:

  1. Data Collection and Annotation:
     • Touch states were automatically annotated using high-frequency AC circuits, resulting in a video dataset containing index finger tapping and swiping data (totaling 211 minutes).
  2. Image Processing:
     • Palm and finger regions were extracted using the MediaPipe hand keypoint model and cropped to a fixed size for input.
     • Image augmentation techniques (e.g., random rotation and brightness adjustment) were applied to enhance model generalization.
  3. Feature Extraction and Classification:
     • Static features: Palm global features and finger local features were extracted using ResNet.
     • Dynamic features: Motion information between frames was extracted using optical flow.
     • All features were input into a classifier via a fully connected layer to distinguish between touch and non-touch states.
  4. Real-Time System Implementation:
     • The system framework was implemented using a multi-process approach, capturing real-time camera data, detecting touch states, and transmitting touch data to MR devices via Socket.

Research Outcomes

• Specific Results:

  1. The touch state detection model using Palmpad achieved a classification accuracy of 97.0% and an F1 score of 96.1% in leave-one-out experiments.
  2. Compared to the LSTM baseline model, Palmpad demonstrated significant advantages in extracting temporal information.
  3. In comparison with existing commercial solutions, Palmpad improved accuracy by 2.8% and spatial resolution by 50%.

• System Advantages:

  1. Compared to traditional depth-based MR gesture tracking, Palmpad offers higher touch state resolution and detection accuracy.
  2. The method does not require additional hardware devices, making it easier to integrate into existing MR headsets.

• Experiments and Evaluations:

  1. Algorithm Performance Evaluation:
     • Palmpad maintained stable performance under different frame rates (e.g., 120fps to 30fps) and supported high-dynamic interactions such as rapid tapping.
     • Ablation experiments confirmed the critical role of optical flow and palm cropping in model performance.
  2. User Experience Study:
     • Compared to existing solutions, Palmpad reduced task completion time for tapping by 3.1%, significantly improving user acceptance and trust.
     • Subjective questionnaires indicated that users found Palmpad easier to use and experienced lower latency.

• Limitations and Future Directions:

  1. System Robustness:
     • Extreme gestures (e.g., fingertip self-occlusion), poor lighting conditions, and high-speed finger movements can affect system stability.
     • Future enhancements could involve multi-camera collaboration, active lighting, or hand markers to improve performance.
  2. Implementation Feasibility:
     • The current hardware prototype requires an external camera; ideally, the system could directly utilize the built-in front camera of MR headsets.
  3. Extended Applications:
     • Support for more forms of palm and finger interactions (e.g., multi-finger operations) and mirrored services for left-handed users.
     • Design of more flexible virtual palm interface mappings to reduce errors caused by palm deformation.

Through the research and implementation of Palmpad, this work expands the boundaries of single-camera MR systems in touch recognition and provides new directions for designing more lightweight MR devices in the future.