Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale

Title of the Paper

Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale

Paper Information

• Domain: Cleaning and classification of mobile user interface (UI) layout data
• Keywords: Dataset, Neural Networks, Mobile UI Layouts, Graph Neural Networks, Transformer, Convolutional Neural Networks

Research Background and Problem

• Existing Issues or Challenges:

  1. Mobile screen UI layout data is often incomplete, including invalid objects, inconsistencies between layouts and screenshots, or overly generic or specific types (e.g., View or ColombiaNativeAdView).
  2. These issues make layout data difficult to utilize in analysis and modeling, potentially reducing model performance.
  3. Traditional manual methods for cleaning layout data are inefficient and costly.

• Significance of the Research:

  • UI layouts are a critical data source for UI design research and semantic understanding. Cleaning and improving layout data is essential for advancing related work and enhancing task performance (e.g., screen summarization and component detection).

• Motivation and Related Work:

  1. Previous researchers have attempted to address these issues using heuristic rules or manual annotation, but these methods suffer from poor scalability and high costs.
  2. The use of deep learning methods to automatically clean layout data is still in its early stages.
  3. This study aims to achieve automated, efficient correction and standardization of layout data using deep learning methods while creating a standardized large-scale dataset.

Solution

• Proposed Method and Solution:

  • A two-stage pipeline called "CLAY" is proposed, leveraging deep learning models to clean layout data.
    1. Invalid Object Detection Module: Removes objects that do not match the screenshot.
    2. Object Type Recognition Module: Assigns a semantically meaningful type to each valid object.

• Innovations:

  1. Replacing manual cleaning with an automated deep learning-based approach, reducing costs and enabling large-scale applications.
  2. Proposing an object type recognition method that combines Graph Neural Networks (GNN) and Transformer to better capture relationships and features between objects.
  3. Creating a higher-quality mobile UI layout dataset, "CLAY," containing 59,555 human-annotated screen layouts.

• Implementation Steps and Key Techniques:

  1. Invalid Object Detection:
     • Utilizes a ResNet-50 convolutional neural network model, combined with a mask channel to distinguish objects for binary classification.
  2. Object Type Recognition:
     • Two models are designed based on Graph Neural Networks and Transformer, respectively.
     • The GNN method leverages the hierarchical tree structure of view relationships through message-passing mechanisms, incorporating pixel, text, and positional information.
     • The Transformer method employs an image-encoded Transformer and applies self-attention mechanisms to model relationships between objects.
  3. Data and Annotation:
     • Cleans and annotates the Rico dataset, filtering 59,555 layouts, marking them as valid or invalid, and defining semantic types for each valid object.

Research Outcomes

• Specific Outcomes:

  1. Developed the CLAY pipeline for automated cleaning and classification.
  2. Produced a high-quality dataset, CLAY, containing 59,555 screen layouts where invalid objects are labeled, and each valid object is assigned a semantically meaningful type.
  3. Model performance:
     • The Invalid Object Detection module achieved an F1 score of 82.7%.
     • The Object Type Recognition module achieved F1 scores of 85.9% (GNN) and 84.7% (Transformer).
     • The new method significantly outperformed heuristic baselines.

• Comparative Analysis and Advantages:

  • The GNN model is more sensitive to common types, while the Transformer performs better with rare categories.
  • Compared to heuristic methods, deep learning approaches are more flexible and exhibit stronger generalization capabilities across different types.

• Experimental or Evaluation Results:

  1. The model demonstrates robustness to rare objects and complex layouts.
  2. Error analysis indicates that the main sources of errors are boundary deviations, overlapping regions, or data imbalance.

• Limitations and Future Directions:

  1. Model training and evaluation are based solely on Android screenshots and have not been validated on other platforms (e.g., iOS).
  2. The approach is still insufficient for certain complex cleaning tasks, such as adding missing objects or modifying attributes.
  3. Future improvements include hybrid model architectures, incorporating full image context information, and building cross-platform datasets.