Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis

Document Title

Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis

Document Information

• Domain: Human-Computer Interaction (HCI), specifically mobile interface design and usability analysis
• Keywords: Mobile UIs, deep learning, explainable artificial intelligence (XAI), vision modeling, tappability, user interface design, machine learning, cluttered visual field analysis, data-driven design, UI usability

Research Background and Issues

• Identified Problems or Challenges:

  • In mobile user interface (UI) design, whether users can accurately perceive if elements are tappable is a critical issue, but existing user studies are time-consuming and costly.
  • While deep learning has been applied to predict tappability, it lacks interpretability, especially in providing designers with clear improvement guidance. Existing methods generally require detailed UI hierarchy data, making them unsuitable for unfinished designs (e.g., wireframes or visual models).
  • Different users have subjective perceptions of UI tappability, posing higher demands on data annotation and model training.

• Significance:

  • The tappability of UI elements directly impacts application usability and user experience (UX). Poor interface design can lead to "false tappability" or low discoverability of essential features.
  • Obtaining accurate and rapid user feedback during the early design stages helps optimize the design process and avoid costly fixes later.

• Research Motivation and Related Work:

  • Related work includes data-driven usability evaluation of UIs, user attention modeling, and deep learning-assisted interface analysis methods. However, these approaches either rely on complex inputs (e.g., view hierarchies, platform-specific features) or lack interpretability in model predictions.
  • Building on these issues, this study aims to overcome the limitations of traditional methods by developing a vision-based tappability model and leveraging machine learning explanation techniques to provide deeper insights into prediction results.

Solution

• Methods and Solutions:

  • Proposed a deep learning model entirely based on visual information to predict the tappability of mobile UI elements, eliminating the need for non-visual information such as view hierarchies.
  • Utilized explainable artificial intelligence (XAI) techniques, including the XRAI algorithm, to generate heatmaps for tappability predictions of UI elements, and employed nearest neighbor search to provide designers with comparative examples that highlight differences in tappability.

• Innovations:

  1. A fully visual, pixel-level model that, compared to existing methods, can be applied to various unfinished designs (e.g., mockups and early prototypes).
  2. Offers local and global levels of explanation mechanisms: local explanations analyze specific input impacts on prediction results via XRAI heatmaps, while global explanations provide designers with inspiration through comparative designs in the dataset.
  3. Developed a tappability dataset with multi-user annotations (extended from the RICO dataset), capturing greater subjective diversity in user perceptions.

• Implementation Steps and Key Techniques:

  • Dataset Construction: Filtered and annotated 18,667 UI elements from the RICO dataset, creating a multi-user (5 annotators per element) dataset on tappability perceptions.
  • Model Construction and Training: Based on the ResNet-18 network, inputs include UI screenshots and binary masks of target regions (indicating prediction areas). Optimization used cross-entropy loss and stochastic gradient descent.
  • Interpretability Analysis: XRAI heatmaps evaluated screen regions relied upon by the model; nearest neighbor search in the model's embedding space generated comparative samples.

Research Outcomes

• Specific Results:

  • Achieved 91.54% accuracy, 80.23% recall, and 0.903 AUC on the test set. Compared to existing methods (Swearngin et al. 2019), the model performed significantly better using only pixel information.
  • Proposed a tappability annotation process aligned with user perceptions, improving subsequent research in training and analysis methods.
  • Successfully demonstrated key influencing factors in UI design using interpretability methods, including text labels, icon proximity, and interface layout.

• Advantages:

  • Compared to traditional methods relying on lengthy text and hierarchical data, this approach expands applicability, enabling instant analysis of incomplete UI implementations.
  • Establishes stronger interpretability links between design and model predictions, offering designers more practical improvement suggestions.

• Experiments and Evaluation Results:

  • Significantly outperformed pixel-input methods in accuracy; while recall was slightly lower compared to more complex schemes combining multiple input features, it maintained a strong comparative advantage.
  • User studies (involving 13 UI/UX professional designers) showed the system significantly reduced design evaluation costs, especially during early iteration stages; some users found the comparative examples generated by the system to be inspiring.

• Limitations and Future Directions:

  • Predictions for large images (e.g., complex ImageViews) may be limited, requiring further adjustments to XRAI heatmap generation.
  • The model focuses on static UI designs and lacks consideration for dynamic contexts in UI workflows.
  • Future improvements include providing more granular design suggestions, adapting to other platforms like iOS, and updating the model with the latest UI datasets to accommodate dynamic changes in UI styles.