Estimating 3D Finger Pose via 2D-3D Fingerprint Matching

Title of the Paper

Estimating 3D Finger Pose via 2D-3D Fingerprint Matching

Paper Information

• Field of Study: Computer Vision and Human-Computer Interaction
• Keywords: 3D finger pose, fingerprint, 3D surface reconstruction, keypoint matching, human-computer interaction

Research Background and Problem Statement

• Identified Problems or Challenges:

  1. Current smart devices primarily rely on 2D touch input, failing to fully utilize the rich information provided by the three-dimensional pose of fingers.
  2. Most finger pose estimation algorithms based on capacitive images only include pitch and yaw angles, neglecting roll angle, and their accuracy is insufficient for large-scale commercial applications.
  3. Additional sensors (e.g., depth cameras or auxiliary sensors) introduce extra costs, making them less suitable for everyday applications.

• Importance of the Problem: Leveraging the 3D pose information of fingers (roll/pitch/yaw) can significantly enrich touch input interaction methods, while improving touch accuracy and user experience.

• Research Motivation and Related Work:

  1. With the advancement of under-display fingerprint sensing technology, obtaining complete fingerprint images enables the calculation of 3D finger poses, creating possibilities for more diverse interaction methods.
  2. There is currently a lack of research combining fingerprint images with 3D reconstruction for finger pose estimation, and publicly available datasets are limited.

Proposed Solution

• Proposed Method or Solution: A novel algorithm for estimating 3D finger pose based on fingerprint images is proposed, with key steps including:

  1. Constructing a fingerprint-specific 3D surface reconstruction model.
  2. Estimating projection errors through keypoint matching between 2D images and 3D point clouds.
  3. Minimizing projection errors to recover three-axis rotation angles.

• Innovations:

  1. For the first time, reconstructed 3D fingerprint surface models are applied to 3D finger pose estimation.
  2. A non-learning analytical method is proposed, reducing reliance on large-scale training datasets and providing strong robustness and generalization capabilities.

• Implementation Steps and Key Techniques:

  1. Fingerprint Surface Reconstruction:
     • Reconstructing the 3D finger surface using sequential fingerprint images.
     • Calculating surface normals via gradient vectors and performing numerical reconstruction using trapezoidal integration.
  2. Pose Estimation:
     • Extracting sparse keypoints from fingerprint images (based on structural features like bifurcation points and endpoints).
     • Performing projection transformations on matched 2D keypoints and 3D model points, minimizing projection errors to solve for finger pose angles (roll, pitch, yaw).
  3. Optimization Scheme:
     • Introducing an accelerated algorithm that combines fast but less stable SURF feature matching with precise keypoint matching to enhance real-time performance and accuracy.

Research Outcomes

• Specific Results:

  1. Experimental results on the authors' constructed 3D finger pose dataset (54 fingers, totaling 23,099 images) show that the method achieves mean absolute errors of 10.74°, 8.25°, and 7.38° for roll, pitch, and yaw estimation, respectively.
  2. The method demonstrates strong robustness, particularly on small touch area sensors.

• Advantages of the Method:

  1. Compared to deep learning models, the method does not rely on large labeled training datasets and exhibits strong adaptability to non-standard regions.
  2. The non-learning method based on simple geometric optimization not only offers high inference efficiency but also significantly reduces dependence on hardware resolution and quality.

• Experimental or Evaluation Results:

  1. Compared to existing baselines (e.g., Gaussian regression or deep learning-based methods), the proposed method outperforms in estimating all three angles, especially roll.
  2. As the touchable area decreases or the fingerprint gallery shrinks, estimation performance shows only slight degradation, validating the algorithm's robustness.

• Limitations and Future Directions:

  1. The resolution and quality of fingerprint acquisition devices directly impact estimation accuracy.
  2. Pre-registration for individual fingers is required, and simplifying this step is necessary for practical applications.
  3. The algorithm currently operates on high-performance configurations; future work should focus on optimizing it for mobile devices.
  4. Further research is needed on real-world user scenarios and potential application functionalities, such as exploring interaction design through user behavior studies.

In summary, this paper provides an innovative method for 3D finger pose estimation, which holds significant importance for touch interaction technology research and practical applications. It also leaves numerous potential improvement paths and open questions for future research.