HandyTrak: Recognizing the Holding Hand on a Commodity Smartphone from Body Silhouette Images

Document Title

HandyTrak: Recognizing the Holding Hand on a Commodity Smartphone from Body Silhouette Images

Document Information

• Subject Area: Human-Computer Interaction, Mobile Computing, Computer Vision
• Keywords: Mobile Computing, Computer Vision, Commodity Smartphone, Deep Learning, Hand Pattern Recognition, User Interface, Real-Time Tracking, Body Silhouette, AI Applications

Research Background and Problem

• Identified Problems or Challenges:

  1. The way users hold their smartphones directly impacts their user experience, especially when one-handed operation is difficult. As smartphone screen sizes increase, it becomes harder for users to reach certain areas of the screen with their thumbs.
  2. Existing methods (e.g., Apple's Reachability feature) require manual activation by the user or can only recognize hand patterns in limited scenarios (e.g., during unlocking), failing to achieve continuous tracking.
  3. Some methods also require additional hardware, making them unsuitable for commodity smartphones.

• Significance:

  1. Improving user experience: Automatically adapting UI layouts to reduce operational difficulty.
  2. Enhancing the efficiency of intelligent interaction, reducing the need for users to manually adjust the UI.

• Research Motivation and Related Work:

  • Existing studies infer hand patterns using input data such as touch events or inertial measurement unit (IMU) sensors, but these require explicit user interaction or rely on additional hardware.
  • The goal of this study is to develop a novel method that leverages the front-facing camera of smartphones to capture users' body silhouettes, enabling continuous tracking without user input.

Solution

• Proposed Solution: HandyTrak is an AI-based software system that uses the built-in front-facing camera of commodity smartphones to capture body silhouette images. Through deep learning algorithms, it continuously identifies whether the user is holding the phone with their left hand, right hand, or both hands.

• Innovations:

  1. No additional hardware is required, relying solely on the smartphone's built-in front-facing camera.
  2. Achieves continuous hand pattern tracking without explicit user interaction.
  3. Proposes a deep learning method for classification using body silhouette images.

• Implementation Steps and Techniques:

  1. Image Preprocessing:
     • Normalize the size of images captured by the front-facing camera (224×224) and perform human segmentation using the FCN-ResNet101 model.
  2. Deep Learning:
     • Design a custom classification network (HandyNet) based on the pre-trained VGG16 model.
     • Classify the segmented silhouette images into left-hand, right-hand, or both-hand categories, with results output through a softmax layer.
  3. System Deployment:
     • HandyTrak is trained and tested in various scenarios, with model parameters fine-tuned through experimentation.

Research Outcomes

• Specific Results:

  1. HandyTrak achieved an average classification accuracy of 89.03% when users held the phone (sampling rate: 2Hz).
  2. For tilted phone-holding scenarios, classification accuracy was higher (95.84%, standard deviation: 5.05%).
  3. HandyTrak achieved 92.3% classification accuracy within a one-second sliding window and performed consistently across different front-facing camera positions.

• Advantages:

  1. Compared to existing methods requiring explicit input or additional hardware, HandyTrak is more convenient and applicable.
  2. It can dynamically adjust UI layouts in smartphone applications, such as adapting virtual button positions during selfies.

• Experimental or Evaluation Results:

  1. The system performed consistently across three common scenarios (standing, sitting, and tabletop support).
  2. Classification accuracy significantly improved in tilted phone-holding scenarios.
  3. Task-specific evaluations showed the following accuracies: 87.56% during unlocking, 88.37% during in-app browsing, and 92.68% during selfies.
  4. Further studies indicated that the system maintained an accuracy of 88.36% even when users were walking.

• Limitations and Future Directions:

  1. Coverage: Currently, the system is trained only for three usage postures (standing, sitting, and tabletop support). Future work should expand to more complex scenarios (e.g., lying down, walking).
  2. User Independence: This study primarily uses user-dependent training models. User-independent models showed weaker performance and require more data for large-scale training.
  3. Energy Consumption and Privacy: Continuous camera usage increases power consumption. Future work could optimize system activation by integrating IMU sensors.
  4. Low-Light Adaptability: The system performs poorly in low-light conditions. Future work could incorporate depth cameras to address nighttime usage.
  5. Further Improvements: Enhanced image segmentation techniques and multimodal data (e.g., depth images) could improve model performance.