Enhancing Smartphone Eye Tracking with Cursor-Based Interactive Implicit Calibration

Research Background and Problem

• Identified Issues or Challenges

  • Existing RGB-based smartphone eye-tracking technologies perform poorly in practical applications, with the following issues:
    1. Lack of personalization, unable to adapt to individual user differences.
    2. Frequent changes in user posture, while eye-tracking calibration is typically only performed during initial setup, making it difficult to maintain high accuracy over time.
    3. Traditional methods require explicit calibration, leading to poor user experience.
  • Optimizing computer vision methods (e.g., expanding datasets or increasing network complexity) can partially improve performance but fails to address posture variation effectively.
  • Some real-time methods (e.g., screen attention-based optimization) have limited effectiveness in modeling complex eye movements in natural environments.

• Significance

  • Eye-tracking technology has tremendous potential for widespread applications on smartphones, such as notification viewing, voice command enhancement, and gaze-based interaction. However, the lack of high-accuracy and low-disruption methods significantly limits its practical use.

• Research Motivation

  • To explore the intrinsic correlation between cursor trajectories and eye movements through low-cost, intuitive hand-eye coordination operations, providing an imperceptible implicit calibration mechanism.
  • To dynamically optimize the eye-tracking model by leveraging data generated during users' natural interactions, thereby improving performance.

Solution

• Method and Advantages

  • A novel interaction-integrated implicit calibration method is proposed — COMETIC (Cursor Operation Mediated Eye-Tracking Implicit Calibration):
    1. Collect calibration data through cursor operations.
    2. Filter reliable cursor positions as substitute data for gaze points based on the correlation between cursor and eye movements.
    3. Continuously fine-tune the model using the filtered data to improve eye-tracking accuracy.
  • Compared to traditional static calibration, COMETIC offers the following advantages:
    1. Eliminates the need for users to interrupt their current tasks for explicit calibration, avoiding disruptions to the user experience.
    2. Dynamically personalizes to adapt to changes in users and usage environments.

• Implementation Steps and Key Techniques

  1. Data Collection: Record facial videos using the smartphone's front camera while simultaneously logging cursor positions and gaze points.
  2. Data Preprocessing:
     • Use image cropping techniques to extract facial and eye regions.
     • Perform statistical analysis to determine the relationship between cursor and gaze points (e.g., cursor trends gradually approaching the target).
  3. Model Design:
     • Model 1: A deep neural network for gaze point extraction from video, using ResNet18 to extract image features and predict gaze positions.
     • Model 2: A model for filtering effective data points from cursor sequences, integrating multi-layer fully connected networks and LSTM.
  4. Fine-Tuning Mechanism:
     • Iteratively fine-tune the above models using data collected during user interactions to optimize performance.
     • Pair effective cursor points with image data for continuous calibration.
  5. Dynamic Interaction Workflow:
     • Users activate the cursor and locate it based on the initial estimated gaze point.
     • Adjust the cursor position by sliding, releasing the finger to complete a "click."

Research Outcomes

• Specific Results

  • Offline Evaluation: COMETIC reduced gaze deviation to 278.3 pixels (1.60 cm, 2.29°) under offline conditions, a 27.2% improvement compared to the non-fine-tuned scenario.
  • Real-Time Evaluation: Achieved a gaze deviation of 446.7 pixels (2.57 cm, 3.68°) in experimental environments, a 50% improvement compared to the non-fine-tuned scenario.

• Advantages Over Existing Methods

  • Compared to traditional explicit calibration-based eye-tracking methods (e.g., nine-point or thirteen-point calibration), COMETIC offers an imperceptible, continuously optimized approach, closer to a "plug-and-play" application scenario.
  • Among implicit methods, the data filtering and calibration based on cursor trajectories demonstrated higher personalization and dynamic adaptability.

• Experimental Results

  • Over five rounds of real-time iterations, gaze error significantly decreased, with the average error dropping from an initial 893.6 pixels to 446.7 pixels after five rounds.
  • The best calibration performance occurred at a distance threshold (𝜏) of 150.0 px, where the error was closest to the actual target point.

• Limitations

  1. Significant Gap Between Real-Time and Offline Performance (37.8% performance gap), attributed to user posture changes and unstable camera content.
  2. Efficiency Issues: Real-time model fine-tuning is time-consuming, typically exceeding 300 seconds.
  3. Privacy Concerns: Model fine-tuning requires uploading user image data to the cloud for computation.
  4. Content Context Influence: The impact of different content scenarios on user gaze behavior has not been sufficiently studied.

• Future Directions

  1. Enhance real-time hardware performance to reduce computational overhead for model training and inference.
  2. Explore multimodal collaborative optimization mechanisms for complex eye movement behaviors.
  3. Optimize modeling and adaptation to user gaze behavior in dynamic and complex scenarios.
  4. Address privacy concerns by researching local encoding techniques or reducing reliance on video details.

Conclusion

COMETIC introduces an implicit calibration method based on cursor operations for smartphone eye-tracking, effectively improving gaze accuracy and interaction experience. While some limitations remain to be addressed, its proposed imperceptible and dynamically optimized strategy provides a practical solution for the widespread application of eye-tracking technology in smart devices.