SUPREYES: SUPer Resolutin for EYES Using Implicit Neural Representation Learning

Document Title

SUPREYES: SUPer Resolution for EYES Using Implicit Neural Representation Learning

Document Information

• Subject Areas: Human-Computer Interaction, Computer Vision, Deep Learning
• Keywords: Eye movement data super-resolution, Implicit neural representation, Upsampling, Self-supervised learning, Human-computer interaction, User identification, Temporal data, Deep learning

Research Background and Problem

• Problem or Challenge:

  • Many current low-resolution eye trackers (e.g., mobile eye-tracking devices) have limited resolution and sampling frequency, restricting the practical utility of eye movement data in many advanced applications.
  • High-resolution eye trackers are expensive, and upgrading hardware involves additional costs such as learning to operate new devices and ensuring hardware compatibility.

• Significance of the Research:

  • High-resolution eye movement data plays a crucial role in applications such as saccade detection, vision-based user interaction, and user identification.
  • Providing an economical and convenient way to enhance the resolution of low-resolution eye movement data can significantly expand the potential of existing hardware and support new application scenarios for eye movement data.

• Motivation and Related Work:

  • Inspired by the performance of Implicit Neural Representation (INR) in tasks like image super-resolution, the authors propose leveraging its ability to model continuous signals for eye movement data super-resolution and generating high-frequency data.
  • Existing work on time-series modeling and super-resolution has not specifically addressed solutions for eye movement data.

Solution

• Method or Solution:

  • SUPREYES: A self-supervised learning framework that maps low-resolution eye movement data into continuous implicit neural representations to enhance the spatiotemporal resolution of the data.
  • Parametric Model: Uses a Multi-Layer Perceptron (MLP) to map input time and local features to eye movement directions.
  • Provides multiple upsampling strategies, including single-stage upsampling and multi-stage upsampling.

• Innovations:

  • Introduced the first learning-driven super-resolution solution for eye movement data.
  • Designed a dedicated global feature extractor for eye movement data and combined it with additional loss functions to optimize model performance.
  • Enforced strict sequence losses (L2 loss) to guide the model in representing saccadic behaviors and preserving individual user characteristics.

• Implementation Steps and Techniques:

  1. Data Preparation: Downsample high-resolution eye movement data to low resolution and use it as input.
  2. Global Feature Extraction: Use a convolutional encoder-decoder (1D CNN) to generate global features from the input.
  3. Local Querying: Query local features and relative time through an MLP to generate eye movement data at the desired resolution.
  4. Loss Construction: Includes positional loss (point error) and reconstruction loss (for global feature extraction).
  5. Evaluation: Assess the quality of generated data using metrics such as Mean Squared Error (MSE) and Synchronized Dynamic Time Warping (sDTW).

Research Outcomes

• Specific Outcomes:

  1. Super-resolution Performance: In tasks upsampling eye movement data from 25 Hz, 50 Hz, and 100 Hz to higher frequencies, SUPREYES outperformed traditional methods such as linear interpolation, PCHIP, and spline interpolation.
  2. User Identification: In user identification tasks based on eye movement data, high-resolution data generated by SUPREYES significantly improved identification accuracy and reduced Equal Error Rate (EER).
  3. Multi-stage Upsampling Effectiveness: Multi-stage upsampling mitigated performance degradation observed in single-stage upsampling and performed better in scenarios with extreme super-resolution challenges.

• Comparison with Existing Methods:

  • Compared to traditional interpolation methods, SUPREYES demonstrated significant advantages in temporal consistency of generated data and retention of user-specific characteristics.
  • In user identification tasks, SUPREYES excelled particularly in downsampling tasks with a scale of ×2.

• Experimental or Evaluation Results:

  • Conducted baseline comparisons and real-world user task experiments on the GazeBase dataset, using metrics such as MAE, MSE, and sDTW to evaluate the differences between generated and real data.
  • In the 50 Hz to 100 Hz super-resolution task, SUPREYES achieved an EER of 2.52%, surpassing all interpolation baselines.

• Limitations and Future Directions:

  • Limitations:

    • The model showed slight underperformance in higher sampling rate super-resolution tasks (e.g., 50 Hz to 200 Hz).
    • The current method may face additional challenges when handling real-world low-resolution eye movement data (e.g., camera hardware noise).
    • The model has not yet achieved real-time operation and is primarily used as a post-processing tool.

  • Future Directions:

    1. Enhance optimization and evaluation for different eye movement types (e.g., saccades, smooth pursuit).
    2. Investigate ways to improve model performance in large-scale upsampling tasks (e.g., unifying feature representations across multiple sampling frequencies).
    3. Explore the potential of SUPREYES in other downstream tasks, such as learning-driven eye movement pattern detection.
    4. Consider integrating the model into real-time low-resolution trackers to enhance sensor sampling capabilities.