ReflecTouch: Detecting Grasp Posture of Smartphone Using Corneal Reflection Images

Document Title

ReflecTouch: Detecting Grasp Posture of Smartphone Using Corneal Reflection Images

Document Information

• Subject Area: Adaptive user interface technology, computer vision, portable smart device interaction
• Keywords: Grasp posture detection, corneal reflection images, smartphone interaction, front-facing camera, convolutional neural network (CNN), user experience optimization

Research Background and Problem

• Identified Problems or Challenges:

  • Users frequently change their grasp posture when using smartphones. Different postures correspond to different operational ranges (e.g., thumb zone, index finger zone), but most applications are designed for right-handed use, which can lead to inconvenience in other scenarios.
  • Current grasp posture detection methods often rely on external sensors (e.g., capacitive sensors or gyroscopes). These methods are hardware-dependent, sensitive to device models, or prone to environmental interference.

• Importance:

  • Grasp posture detection can support adaptive smart interfaces, optimizing screen content or button positions to enhance user experience. This is particularly critical for large-screen smartphones to adapt to users' actual operational ranges.

• Research Motivation and Related Work:

  • Existing grasp posture detection methods often rely on additional hardware, such as microphones or accelerometers, which increase costs and are highly device-dependent.
  • The authors observed that the smartphone screen creates a reflection in the user's cornea, proposing to detect grasp posture by capturing this reflection using the front-facing camera, thereby eliminating the need for additional hardware.

Solution

• Method and Solution:

  • A novel method, ReflecTouch, is proposed. It captures the reflection of the smartphone screen in the user's cornea using the front-facing camera and employs a convolutional neural network (CNN) to analyze whether fingers are occluding the reflection and to identify six types of grasp postures.
  • This method requires no additional hardware and can be implemented on existing smartphones.

• Innovative Aspects:

  • Utilizes the image information of the smartphone screen reflected in the cornea for grasp posture detection.
  • Minimal dependency on smartphone models and no need for additional devices.
  • The detection process relies solely on the built-in front-facing camera, reducing hardware costs.

• Implementation Steps and Key Techniques:

  1. Data Collection: Use a smartphone application to capture facial images of users in different grasp postures, including the screen reflection area.
  2. Data Preprocessing:
     • Extract the eye reflection region of interest (ROI).
     • Adjust image brightness and contrast, and augment data to enhance diversity.
  3. Model Training and Prediction:
     • Train a CNN model using the preprocessed data, with input as 80×80×2 reflection images (left and right eyes) and output as probability estimates for six grasp postures.
     • Split samples into training and testing datasets, and optimize the model using various training strategies (e.g., all-user learning, cross-validation with other users, transfer learning).

Research Findings

• Specific Results:

  • Validation experiments demonstrated that ReflecTouch achieved an average recognition accuracy of 85% for six grasp postures (all-user learning method).
  • Compared to other sensor-based or alternative methods, this approach requires no additional hardware and is compatible with existing smartphone sensors, ensuring device universality.

• Advantages Over Existing Solutions:

  • No external sensors: Unlike many solutions that rely on capacitive sensors or microphones, this method solely depends on the smartphone's built-in front-facing camera.
  • Avoids hardware dependency: Not affected by differences in sensor placement or specifications across smartphone models.
  • Expands the usability of smartphone grasp posture detection, especially in scenarios involving transitions between single-handed and two-handed operations.

• Experimental or Evaluation Results:

  • All-User Learning: Accuracy of 85%.
  • Other-User Learning (cross-validation method): Accuracy of 41%, limited by variations in users' eye biological characteristics.
  • Transfer Learning: Using a per-user transfer learning approach, the accuracy reached 76% with only 24 training images.

• Limitations and Future Directions:

  • Lighting Issues: The method is sensitive to lighting conditions, and strong light may hinder accurate detection of reflection images.
  • Angle Constraints: Current experiments require users to face the camera directly, rather than allowing free-angle usage.
  • Real-Time Challenges: Computational costs are relatively high for real-time prediction. Future work should optimize the algorithm or reduce input image resolution.
  • Application Scalability:
    • Enable detection of more complex gestures, such as distinguishing between different finger touches or predicting mid-gesture transitions.
    • Combine with other sensors to improve recognition accuracy in specific scenarios (e.g., horizontal screen gestures).