Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning

Document Title

Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning

Document Information

• Subject Area: User Interface (UI) Understanding and Multimodal Learning, Automated Description Generation with Deep Learning
• Keywords: Mobile UI summarization, screen understanding, deep learning, language-based UI, dataset

Research Background and Problem

• Identified Problem or Challenge: Current mobile user interfaces (UIs) lack language summarization capabilities, making it difficult to quickly understand screen content through natural language. This poses significant barriers for language-based application scenarios.
• Importance of the Problem: Automatically generating language summaries can help users quickly grasp screen content and functionality, particularly for visually impaired users or language interaction applications. This task also expands the potential for integrating language with user interfaces, promoting the adoption of multimodal interaction.
• Motivation and Related Work:
  • Existing work primarily focuses on describing individual components (e.g., textual descriptions of icons) and cannot generate holistic descriptions of entire screens.
  • Screen summarization requires comprehensive understanding of multimodal data (images, text, structure, UI semantics), which remains an unresolved challenge.
  • Data-driven deep learning has demonstrated broad potential in multimodal data encoding and task scenarios (e.g., UI retrieval), but research on generating screen summaries adapted to natural language expression is still scarce.

Solution

• Method and Solution: The Screen2Words framework is proposed, leveraging multimodal learning and deep models to generate language-based screen summaries.
• Innovations:
  • Developed the first large-scale screen summarization dataset containing 112,085 human-annotated language summaries (covering 22,417 Android screens), ensuring data quality and consistency.
  • Designed a dual-modal encoding architecture based on Transformer and ResNet, integrating visual, structural, and semantic information of screens to achieve summarization.
• Implementation Steps and Techniques:
  1. Dataset Creation: Selected 22,417 screens from the publicly available Rico dataset and annotated them with language summaries by professional labelers.
  2. Model Architecture: Designed a deep learning-based Transformer+ResNet dual-modal model, comprising two parts to encode screen structure and image information.
     • Encoding Layout and Text Information: Used Transformer to encode UI structure and screen text.
     • Encoding Image Information: Used ResNet to process images of screen elements.
  3. Training and Evaluation: Performed end-to-end training using cross-entropy loss and generated screen summaries through Beam Search.
  4. Model Variants: Tested the effects of different modality combinations (images only, images+layout, text+images+layout, etc.).
  5. Baseline Comparison: Designed heuristic methods (e.g., TF-IDF and pixel similarity) as template-matching baselines for performance validation.

Research Outcomes

• Specific Results:
  • The deep learning method significantly outperformed previous template retrieval baselines across all automated evaluation metrics.
  • The complete model (Pixel+Layout+Text+AppDesc) achieved the best performance on BLEU, CIDEr, ROUGE-L, and METEOR metrics.
  • Human evaluation results also confirmed that the model-generated summaries were concise and accurate (average score of 3.436, higher than baseline scores).
• Advantages:
  • Compared to traditional template retrieval methods, Screen2Words more accurately captures UI semantics.
  • The integration of multimodal data (visual, structural, textual) effectively enhances summary quality.
• Experiments and Evaluation Results:
  • Different model variants demonstrated dependency on multimodal inputs: the complete model significantly outperformed models based on single modalities (e.g., images only).
  • Automated evaluation results aligned with human subjective ratings, indicating high consistency and detailed performance in the model-generated summaries.
• Limitations and Future Directions:
  • Generated results may still occasionally produce overly generic or incorrect descriptions.
  • Currently supports only full-screen descriptions, not regional or targeted descriptions; future work could extend to visual question answering (VQA).
  • Performance may be limited in scenarios lacking text or UI structure information; future optimization is needed to address multimodal data deficiencies.

Application Scenarios and Future Work

• Application Scenarios:
  1. Language-Based UI Retrieval: Search screen design examples through natural language descriptions, enabling more flexible and detailed queries.
  2. Enhanced Screen Reader Functionality: Provide visually impaired users with quick screen overviews, helping them efficiently locate and understand content.
  3. Screen Indexing and Conversational Interaction: Quickly access specific screens via voice commands, enabling semi-automated task completion in conjunction with multimodal conversational agents.
• Future Directions:
  • Expanding to Language-Generated UI Design: Utilize screen summaries to generate mobile UI design layouts.
  • Developing UI-Based Question-Answering Systems: Address user queries about specific screen regions, enhancing UI language interaction capabilities.
  • Improving Models and Evaluation Methods: Enhance the precision of screen summaries and design evaluation methods more aligned with human subjective perception.

Open Data and Community Support

• Screen2Words has released its dataset and model source code to provide a data-driven foundation for UI language research and promote the development of related applications.