Never-ending Learning of User Interfaces

Title of the Paper

Never-ending Learning of User Interfaces

Paper Information

• Research Domain: Application of human-computer interaction and machine learning models in UI automation
• Keywords: User interface, UI modeling, machine learning, application crawling, continuous learning

Research Background and Problems

• Identified Issues or Challenges:
  • Many current machine learning models rely on manually labeled static screenshot datasets, which are costly to collect and prone to errors. Tasks such as determining whether a UI element is tappable often suffer from labeling uncertainty.
  • Traditional datasets fail to reflect real-world scenarios after dynamic updates of applications.
• Importance:
  • Automating the acquisition of UI semantic data and continuously updating models can improve application accessibility, testing efficiency, and automation, thereby better supporting users.
• Research Motivation and Related Work:
  • Existing datasets like Rico and AMP provide large-scale mobile UI data, but most are generated through manual interaction or static analysis.
  • Never-ending Learning, first applied in knowledge extraction, has the potential to be extended to UI modeling to address dynamic data collection challenges.

Solution

• Proposed Method or Solution:
  • Develop an automated crawling system—Never-ending UI Learner—that installs real applications from mobile app stores, discovers semantic attributes of UI elements through interaction, generates labeled data, and continuously updates machine learning models.
• Innovations:
  • Fully automated data collection and labeling without human supervision.
  • The crawler dynamically adapts to application updates and design trends.
  • After initial training with manually labeled data, the model can be updated entirely with new interaction data, eliminating the need for manual labeling.
• Implementation Steps and Key Technologies:
  • System Architecture:
    • A coordination server manages the application list and distributes tasks to multiple worker nodes.
    • Worker nodes automatically install applications on mobile devices, interact with them, and generate data.
  • Data Processing and Model Design:
    • The crawler records time-series screenshots of interactions and heuristically labels interaction results, such as tappability and draggability.
    • Deep neural network models predict UI element semantics, comprising two main components:
      1. A CNN-based screen similarity model.
      2. A UI element detection and semantic prediction model (e.g., tappability and draggability).

Research Results

• Specific Outcomes:
  • The crawler system ran for over 5000 hours, performed more than 500,000 operations on 6461 applications, and generated a labeled dataset larger than existing datasets.
  • Successfully trained three UI semantic models:
    • Tappability: Predicts whether an element is tappable, achieving a final F1 score of 0.860.
    • Draggability: Predicts whether an element is draggable, achieving a final F1 score of 0.794.
    • Screen Similarity: Predicts page similarity, achieving a final F1 score of 0.663.
• Advantages Compared to Existing Solutions:
  • Significantly reduces the cost of manual labeling while improving labeling reliability.
  • Stronger adaptability to dynamic design changes in real-world application scenarios.
• Experimental and Evaluation Results:
  • Compared random crawling and uncertainty sampling strategies, with random crawling achieving optimal performance.
  • All models showed rapid performance improvement during early training stages, with performance improvements plateauing in subsequent crawling cycles.
  • Fine-tuning the screen similarity model with crawler-generated data achieved better results than training solely on the original dataset.
• Limitations and Future Directions:
  • The system currently supports crawling and analysis of iOS applications only; future work could extend to other platforms (e.g., Android and Web).
  • Heuristic methods for interaction labeling may not generalize effectively to certain complex semantics.
  • Improvements are needed for handling data redundancy issues (e.g., repeated crawling of similar scenarios).
  • Explore learning methods for additional UI semantics, such as "press-and-hold" functionality or text box input behavior.

Conclusion

This study implements a continuous learning framework based on automated interaction for dynamically collecting and modeling UI semantic data, featuring high automation and scalability. The approach not only effectively reduces the cost of manual labeling but also learns and adapts to evolving UI design trends. In the future, it is expected to have a significant impact on improving application accessibility, automated testing, and UI usability enhancement.