Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots

Title of the Paper

Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots

Paper Information

• Subject Area: Reverse engineering and model parsing of user interfaces (UI)
• Keywords: UI modeling, UI semantics, hierarchy prediction, accessibility enhancement, code generation

Research Background and Problem

• Problem or Challenge: Current methods for automatic understanding of user interfaces often rely on metadata provided by developers, such as view hierarchies or accessibility labels. However, this metadata is often missing or incomplete. Additionally, many existing methods face limitations in grouping UI elements semantically and constructing structured interface definitions.
• Significance: Automatically parsing UI structures can enhance application accessibility, support task automation, improve interface design efficiency, and reduce manual intervention.
• Motivation and Related Work:
  • The research explores how to parse hierarchical structures from UI screenshots to achieve UI understanding.
  • Inspired by parsing techniques in natural language processing, the study proposes innovative solutions for UI screen parsing.

Solution

• Proposed Solution: Screen Parsing, which predicts structured models of UI elements and their relationships from screenshots.
• Innovations:
  • Applying natural language parsing techniques to UI modeling, using machine learning to generate UI hierarchy structures.
  • The method includes UI element detection, UI hierarchy prediction, and element grouping classification.
• Implementation Steps and Techniques:
  1. UI Element Detection: Using the Faster-RCNN model to extract UI elements and their parameters, such as position and type, from the screen.
  2. UI Hierarchy Prediction: Employing a stack-based top-down parser and an LSTM model to predict the hierarchical relationships of UI elements and generate a tree structure.
  3. Grouping Classification (Labeling): Using a Deep Averaging Network (DAN) to classify intermediate nodes with labels such as button groups, list items, toolbars, and other common container types.

Research Outcomes

• Specific Outcomes:
  • Developed and implemented a screen parsing model that successfully predicts UI hierarchy structures.
  • Achieved higher accuracy on various performance metrics compared to baseline methods (e.g., Screen Recognition), with improvements of up to 23%.
• Advantages Compared to Existing Solutions:
  • Supports a broader range of applications, such as structural similarity computation, enhanced screen reader accessibility, and code generation.
  • The hierarchy prediction method provides a global perspective, avoiding reliance on local heuristic grouping.
• Experimental or Evaluation Results:
  • The proposed dynamic training method (Dynamic Oracle) significantly outperformed traditional static training methods (Static Oracle).
  • Comprehensive performance metrics, including F1 score, Tree Edit Distance (GED), and Clustering Matching Score (CM), showed superior results on AMP (iOS dataset) and RICO (Android dataset) compared to baseline methods.
  • Demonstrated stable performance on complex screens, maintaining competitiveness even with a large number of screen elements.
• Limitations and Future Directions:
  • Screen parsing is susceptible to the performance of the UI element detection model. Future work could improve performance with more robust object detection techniques.
  • The current method primarily handles single screenshots, whereas certain UI semantics (e.g., task models) require processing screen sequences.
  • The scope of screen parsing could be expanded to include data flow, interaction, and task model extraction.

Example Applications

1. UI Similarity Search:
   • Generating UI embedding vectors from the parsing model's intermediate representations to measure screen similarity.
   • Application scenarios: app crawling, information extraction, and semantic structure analysis.
2. Accessibility Enhancement:
   • Improving the navigation order of screen readers to help visually impaired users use interfaces more efficiently.
   • The parsing model addresses limitations of traditional heuristic methods, such as handling global information.
3. UI Code Generation:
   • Generating code (e.g., SwiftUI) from screenshots to support UI prototyping.
   • The output code supports responsive layouts, adapting to different screen sizes and device types.

Additional Information

• Experimental Datasets: AMP (130,000 iOS screens), RICO (80,000 Android screens).
• Main Model Architecture and Hyperparameter Settings: Faster-RCNN for element detection, stack-based parser for hierarchy prediction, and Deep Averaging Network for grouping classification.
• Examples of model shortcomings include: object detection errors, grouping errors, and grouping omissions.