Bridging Gulfs in UI Generation through Semantic Guidance

Paper Title

Bridging Gulfs in UI Generation through Semantic Guidance

Publication Info

• Topic area: Generative AI for user interface (UI) design
• Keywords: Generative AI, UI design, semantic representation, intent specification, iterative refinement, execution gulf, evaluation gulf, design semantics, user study, AI transparency

Background and Problem

• Problem / challenge: Current generative UI systems face significant challenges in enabling users to articulate design intent, interpret generated outputs, and refine designs iteratively. These challenges reflect Norman’s gulfs of execution (difficulty expressing intent) and evaluation (difficulty interpreting results), which are amplified through iterative refinement.
• Significance: Addressing these challenges is critical for making generative UI systems more usable, predictable, and effective, particularly for professional design workflows where precision and control are paramount.
• Motivation and related work: Prior work has explored rule-based and data-driven UI generation, intent specification tools, and semantic representations. However, existing systems typically focus on either input structuring or output evaluation, without connecting the two in a unified framework. This paper addresses this gap by introducing a semantic intermediate layer that bridges intent specification and output evaluation.

Solution

• Proposed approach: A semantic-based UI generation system that introduces a four-level hierarchical framework (Product, Design System, Feature, Component) to structure design semantics and serves as an intermediate layer between user intent and AI-generated outputs.
• Novelty:
  1. Development of a hierarchical framework of design semantics for UI generation.
  2. Introduction of semantic representations to bridge the gulf between user intent and generated outputs.
  3. Implementation of a system that supports structured specification, transparent analysis, and relationship-aware refinement.
• Procedure and key techniques:
  • Conducted thematic analysis of prompting guidelines from six major UI generation services to derive the hierarchical semantic framework.
  • Designed a web-based system that uses the framework for structured input, semantic extraction, and iterative refinement.
  • Integrated GPT-5 for semantic parsing and analysis, and Vercel’s v0-1.5-md for UI generation.
  • Implemented features such as semantic relationship visualization, scoped editing, and augmented semantic analysis to improve transparency and control.

Results

• Concrete findings:
  • The semantic system significantly improved intent expression (mean score: 5.93 vs. 4.64), output interpretability (5.93 vs. 3.86), and ease of modification (6.00 vs. 3.71) compared to a baseline chat-based system.
  • Participants reported reduced semantic drift and more predictable iterative refinement.
• Advantage over baselines:
  • Higher ratings across all measured dimensions, including intent expressiveness, output transparency, and iterative controllability.
  • Enhanced user trust and understanding of AI decisions through semantic extraction and relationship analysis.
• Experiments / evaluation:
  • Conducted a controlled user study with 14 participants (UI/UX designers, software engineers, product managers) using a within-subjects design to compare the semantic system with a chat-based baseline.
  • Tasks included target screen generation and open-ended screen design, with a minimum of three iteration cycles per task.
  • Quantitative measures (7-point Likert scales) and qualitative interviews were used to evaluate system effectiveness.
• Limitations and future work:
  • Initial learning curve for semantic categories and potential nuance loss in parsing long-form intents.
  • Limited evaluation scope; future studies should examine team-level workflows and multi-screen designs.
  • Technical challenges include scaling to complex UI architectures and improving model reliability for localized edits.

Summary

This paper introduces a semantic-based system for generative UI design that bridges the gulfs of execution and evaluation by structuring design semantics into a four-level hierarchical framework. The system enables users to specify intent, analyze outputs, and refine designs iteratively with greater transparency and control. A user study demonstrated significant improvements in intent articulation, output interpretation, and iterative refinement compared to a chat-based baseline. While the approach introduces some initial learning overhead, it provides a systematic and interpretable workflow for AI-driven UI design, with potential applications in professional and collaborative settings. Future work will focus on scaling the approach to more complex and creative design scenarios.