UEyes: Understanding Visual Saliency across User Interface Types

Title of the Paper

UEyes: Understanding Visual Saliency across User Interface Types

Paper Information

• Research Area: Visual saliency prediction in computer vision and human-computer interaction
• Keywords: Human perception and cognition, interaction design, computer vision, deep learning, eye-tracking

Research Background and Problem Statement

• Problems and Challenges:

  • Different types of user interfaces (UIs) exhibit significant variations in layout and use of visual elements, yet there is limited understanding of how these differences influence user gaze patterns.
  • Existing datasets are often small in scale or limited to specific UI types, and many studies rely on mouse tracking or manual annotations as proxies for eye-tracking data, which compromises accuracy.
  • Current models for visual saliency and gaze path prediction perform poorly when trained uniformly across different UI types.

• Significance of the Research:

  • Understanding visual saliency characteristics in UIs is crucial for optimizing design and enhancing user experience.
  • Data-driven visual saliency models can guide UI designers in effectively balancing the distribution of visual information.

• Research Motivation:

  • To propose and analyze an eye-tracking dataset covering multiple UI types, revealing differences in visual saliency across these types.
  • To improve existing saliency prediction models to better adapt to cross-UI scenarios.

Solution

• Overview of the Solution:

  • Developed a novel high-fidelity eye-tracking dataset, UEyes, encompassing four major UI types: web, desktop UI, mobile UI, and posters, with 1,980 UI screenshots and gaze paths from 62 users.
  • Analyzed visual saliency characteristics (e.g., positional bias, color bias, scan path angles, and amplitudes).
  • Compared various saliency prediction models and developed improved models (UMSI++ and SAM++) to enhance performance.

• Key Innovations:

  1. Introduced and publicly released a generalized dataset, UEyes, featuring diverse UI types and high-quality eye-tracking data.
  2. Systematically analyzed visual saliency differences across UI types, revealing unique biases such as upper-left region preference and text prioritization.
  3. Proposed more effective loss functions and training strategies to optimize saliency prediction models for cross-UI applicability.

• Implementation Steps and Key Techniques:

  • Data Collection: Gathered user gaze data using high-fidelity eye-tracking equipment (60 Hz sampling rate).
  • Data Analysis: Conducted statistical comparisons of saliency maps, gaze paths, and specific attributes (e.g., color, region).
  • Model Improvement: Developed new saliency prediction models (UMSI++) by integrating distributed loss (KL divergence) and position-based loss (NSS).
  • Model Evaluation: Compared multiple saliency and gaze path prediction models, including traditional computational models, data-driven approaches, and improved models.

Research Outcomes

• Specific Outcomes:

  • The UEyes dataset includes 495 screenshots of web, desktop UI, mobile UI, and posters, along with their associated eye-tracking data.
  • Analysis revealed a strong upper-left region gaze bias across all UI types, with text elements attracting more attention than images.
  • Identified distinct characteristics for different UI types, such as fewer salient elements and higher user focus in mobile UIs, and longer horizontal gaze paths in posters.

• Model Performance Improvements:

  • The optimized UMSI++ model improved the prediction accuracy of saliency maps across different UI types, outperforming competing methods in standard AUC metrics.
  • Results demonstrated that training on multi-type UIs significantly enhances model generalization and cross-domain prediction performance.

• Experimental and Evaluation Results:

  • The proposed models outperformed existing models across various evaluation metrics (AUC, NSS, KL divergence, etc.).
  • In gaze path prediction analysis, UMSI++ and DeepGaze++ models captured multi-dimensional gaze characteristics, though further optimization is needed for path accuracy.

• Limitations and Future Directions:

  1. The realism of mobile UI scenarios in the dataset needs improvement; future work should focus on real-world mobile device usage environments.
  2. Current analyses rely on broad element categories (e.g., text, images); future research could incorporate semantic classification and user task objectives.
  3. Saliency maps generated by the models still exhibit numerous false-positive hotspots; future work should enhance suppression of non-salient regions.
  4. Gaze path prediction models struggle to capture gaze sequence accurately; introducing more complex temporal embeddings and behavioral pattern modeling is necessary.

Conclusion

This study systematically explored the characteristics and variations of user visual saliency across multiple UI types for the first time. The newly proposed UEyes dataset and improved models (UMSI++, SAM++) provide a solid foundation for future intelligent design tools. The research highlights the need for higher-fidelity data, richer interaction contexts, and task objective information to further improve the predictive capabilities and design guidance of saliency and gaze path models in complex UI scenarios.