MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling

Document Title

MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI Modeling

Document Information

• Domain: Mobile User Interface (UI) research in the field of Human-Computer Interaction (HCI)
• Keywords: Dataset, UI modeling, large language models, mobile user interface, noise filtering, automated exploration, user interface retrieval, element detection

Research Background and Problems

• Problems and Challenges:

  • Mobile UI modeling requires high-quality datasets, but existing datasets are often outdated (e.g., the Rico dataset has not been updated since 2017), contain noisy data (e.g., inaccurate view hierarchies), and fail to meet the requirements of modern UI design.
  • Noise includes partially rendered UIs, overlapping view hierarchies, and duplicate screens, which may degrade the performance of data-driven UI modeling.
  • Models trained on outdated UIs exhibit performance degradation when dealing with modern designs.

• Importance:

  • Mobile applications are increasingly significant, and UI modeling involves multiple tasks (e.g., interface element detection, screen embedding, control label generation, design search, etc.).
  • The lack of high-quality, modern-style UI datasets limits the development of these tasks.

• Motivation and Related Work:

  • Existing datasets (e.g., Rico) feature outdated design styles and significant noise issues, impacting the performance of data-driven modeling.
  • The semantic understanding capabilities of large language models (LLMs) provide new possibilities for automated application exploration.
  • A mobile UI dataset with modern styles, high-quality view hierarchies, and noise-free data is urgently needed.

Solution

• Proposed Method:

  1. Introduce a novel LLM-enhanced automated application exploration method, framing application exploration as a question-answering task, where the LLM simulates human exploration of mobile applications.
  2. Employ mature techniques and best practices to filter noisy data, including duplicate detection based on view hierarchy data and deep learning methods to detect overlapping view hierarchies.
  3. Perform manual verification after automated filtering to ensure the final quality of the dataset.

• Innovations:

  • First application of LLMs to mobile application exploration, simulating expert human behavior to trigger a broader range of UIs.
  • Combination of denoising techniques and manual verification significantly improves dataset quality.
  • Collection of a high-quality, modern-style mobile UI dataset based on popular recent applications.

• Implementation Steps and Techniques:

  1. Application Collection: Develop a cloud-based web crawler to obtain popular applications from the Google Play Store.
  2. Automated Exploration: Use LLMs to generate user interaction scripts and execute actions via Android tools (e.g., ADB, UIAutomator).
  3. Noise Validation and Filtering:
     • Remove duplicate data using the MD5 hashing method.
     • Filter overlapping view hierarchies using deep learning models.
     • Conduct manual reviews to ensure data quality.
  4. Data Analysis and Release: Collect 18,000 UIs, each containing screenshots and high-quality view hierarchies.

Research Outcomes

• Specific Results:

  • Constructed the MUD dataset, comprising 18,132 unique UI screens from 3,300 Android applications (covering 33 application categories).
  • The dataset was used for two tasks: UI element detection and UI retrieval.
  • Experimental results demonstrate that MUD outperforms existing datasets (e.g., Rico) in modernity and quality.

• Advantages:

  • Automated exploration achieved 17% higher coverage compared to three existing tools (Monkey, Droidbot, Humanoid), significantly improving exploration comprehensiveness.
  • In UI element detection, MUD improved the average precision (AP) by 10.5% compared to Rico, especially excelling in modern UI elements.
  • In the UI retrieval task, MUD achieved a relevance score of 4.1 (out of 5), significantly higher than Rico’s score of 3.2.

• Experiments and Evaluation Results:

  • The LLM-enhanced exploration method demonstrated high capabilities in application coverage, effective text input, composite operation generation, and multilingual support.
  • The dataset’s UI quality was refined through noise filtering and manual verification, retaining approximately 40% of the optimal data.
  • Experiments show that MUD significantly outperforms the Rico dataset in two downstream tasks.

• Limitations and Future Directions:

  • Limitations:

    • The current dataset only includes Android applications, excluding iOS and web application data.
    • Screenshots and view hierarchies lack coverage of interaction tracking and multimodal information (e.g., animations).
    • LLM exploration excludes complex account registration processes.

  • Future Directions:

    • Expand to other platforms (e.g., iOS, web) to construct a multimodal mobile UI dataset.
    • Improve LLM models to leverage more contextual information (e.g., app descriptions and interaction history) for enhanced exploration efficiency.
    • Design comprehensive account registration/login solutions to overcome the limitations of existing exploration tools.
    • Extend the dataset to support more modeling tasks (e.g., screen question answering, screen summarization) to further validate its broad applicability.

Conclusion

This paper proposes an innovative automated method to address the issues of outdated and noisy UI datasets, constructing the MUD dataset with over 18,000 modern-style UIs. Experiments validate the advantages of the method and the effectiveness of the dataset, paving the way for new advancements in UI modeling research. The dataset has been publicly released, laying a foundation for promoting high-quality UI design research.