UISketch: A Large-Scale Dataset of UI Element Sketches

Title of the Paper

UISketch: A Large-Scale Dataset of UI Element Sketches

Paper Information

• Subject Area: User Interface (UI) Design, Artificial Intelligence (AI), Computer Vision
• Keywords: Dataset, UI Element Sketches, Sketch Recognition, Deep Neural Networks (DNN), Low-Fidelity Prototypes, Wireframes, Human-Computer Interaction Design

Research Background and Problem

• Identified Challenges:
  • There is currently a lack of large-scale datasets of hand-drawn user interface (UI) element sketches to support the automation of low-fidelity (lo-fi) UI design-to-code conversion using deep learning models.
  • There is a lack of studies directly comparing the accuracy of human and machine recognition of UI element sketches.
• Significance:
  • UI designers often use sketches as a tool for quickly expressing and communicating design ideas, and low-fidelity hand-drawn sketches are a key resource in the early stages of UI design.
  • Automating the transformation of hand-drawn sketches into high-fidelity designs or code can significantly improve design efficiency and reduce design time.
• Research Motivation and Related Work:
  • Recent studies have begun exploring the use of deep learning to automatically convert hand-drawn sketches into code, but there is a lack of large-scale annotated datasets of UI sketches.
  • Existing datasets (e.g., RICO, TU Berlin Sketch Dataset) primarily focus on screenshots or object sketches, which do not meet the specific needs of UI element sketch recognition.

Solution

• Proposed Method:
  • The authors created the UISketch dataset, which includes 17,979 hand-drawn UI element sketches covering 21 UI element categories (e.g., buttons, dropdown menus, text boxes).
  • The dataset was sourced from 967 participants, including UI/UX designers, front-end developers, and graduate students in Human-Computer Interaction (HCI) and Computer Science (CS).
  • Participants provided sketches via paper-based or digital questionnaires (using pens, pencils, or styluses).
• Innovations:
  • UISketch is the first large-scale open dataset of UI element sketches and is the first to study human versus machine sketch recognition capabilities based on this dataset.
  • Deep neural networks (DNNs), such as ResNet-152, were used to classify the sketches, and the results were compared with human recognition performance.
• Implementation Steps and Key Techniques:
  1. Design sketch questionnaires containing 21 UI elements as drawing samples.
  2. Collect sketches and digitize paper-based sketches using techniques such as Otsu thresholding.
  3. Train and evaluate 26 existing deep learning models (e.g., ResNet, DenseNet, VGG) on the dataset.
  4. Use t-SNE visualization tools to analyze the clustering of sketch features by deep models.

Research Outcomes

• Specific Results:
  • Created the UISketch dataset comprising 17,979 hand-drawn sketches, which is publicly available for the research community.
  • Human recognition accuracy for UI sketches was approximately 96.49%, while the best-performing deep learning model (ResNet-152) achieved a recognition accuracy of 91.77%.
  • Found that computers struggle with structural similarity in UI elements, while human recognition errors are more influenced by semantic similarity.
• Comparative Advantages:
  • UISketch provides an important benchmark for future research on sketch-based image retrieval, automated sketch completion, and sketch-to-code conversion.
  • The dataset includes digitized sketches and their approximate average images, aiding in the understanding of common structural features of UI elements.
• Experimental and Evaluation Results:
  • ResNet-152 performed best in structure-based classification but still fell short of human recognition accuracy.
  • t-SNE clustering results showed that models tend to classify sketches based on structural similarity without capturing semantic context.
• Limitations and Future Directions:
  • The dataset currently only includes common UI elements for mobile devices (Android). Future plans include expanding to UI elements for web and desktop platforms.
  • Future research aims to collect more semantically rich and timestamped vector sketches to enhance the semantic understanding of models.
  • Plans to build a comprehensive dataset of low-fidelity UI screen sketches and an annotated object detection dataset.

Conclusion

• The UISketch dataset paves the way for research on AI-assisted UI design, including sketch completion, sketch classification, and code generation.
• While deep learning performance is approaching human levels, further improvements are needed by integrating semantic understanding. This study is the first to systematically analyze the structural and semantic characteristics of UI sketches and their recognition mechanisms.