Mapping the Design Space of User Experience for Computer Use AgentsLarge language model (LLM)-based computer use agents execute user commands by interacting with available UI elements, but little is known about how users want to interact with these agents or what design factors matter for their user experience (UX). We conducted a two-phase study to map the UX design space for comput…2026RCRuijia Cheng et al.AppleHuman-LLM CollaborationExplainable AI (XAI)AI-Assisted Decision-Making & AutomationIUI
Athena: Intermediate Representations for Iterative Scaffolded App Generation with an LLMIt is challenging to generate the code for a complete user interface using a Large Language Model (LLM). User interfaces are complex and their implementations often consist of multiple, inter-related files that together specify the contents of each screen, the navigation flows between the screens, and the data model u…2026JBJon-Tait Beason et al.AppleHuman-LLM CollaborationPrototyping & User TestingComputational Methods in HCIIUI
The Way We Notice, That’s What Really Matters: Instantiating UI Components with Distinguishing VariationsFront-end developers author UI components to be broadly reusable by parameterizing visual and behavioral properties. While flexible, this makes instantiation harder, as developers must reason about numerous property values and interactions. In practice, they must explore the component’s large design space and provide…2026PVPriyan Vaithilingam et al.AppleHuman-LLM CollaborationPrototyping & User TestingComputational Methods in HCICHI
Improving User Interface Generation Models from Designer FeedbackDespite being trained on vast amounts of data, most LLMs are unable to reliably generate well-designed UIs. Designer feedback is essential to improving performance on UI generation; however, we find that existing RLHF methods based on ratings or rankings are not well-aligned with with designers' workflows and ignore t…2026JWJason Wu et al.Carnegie Mellon UniversityHuman-LLM CollaborationPrototyping & User Testing360° Video & Panoramic ContentCHI
SQUIRE: Interactive UI Authoring via Slot QUery Intermediate REpresentationsFrontend developers create UI prototypes to evaluate alternatives, which is a time-consuming process of repeated iteration and refinement. Generative AI code assistants enable rapid prototyping simply by prompting through a chat interface rather than writing code. However, while this interaction gives developers flexi…2025ALAlan Leung et al.AppleHuman-LLM CollaborationKnowledge Worker Tools & WorkflowsUIST
From Interaction to Impact: Towards Safer AI Agent Through Understanding and Evaluating Mobile UI Operation ImpactsWith advances in generative AI, there is increasing work towards creating autonomous agents that can manage daily tasks by operating user interfaces (UIs). While prior research has studied the mechanics of how AI agents might navigate UIs and understand UI structure, the effects of agents and their autonomous actions—…2025ZZZhuohao (Jerry) Zhang et al.University Of WashingtonGenerative AI (Text, Image, Music, Video)AI-Assisted Decision-Making & AutomationIUI
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine ConversationsMultimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data. In this paper, we adapt a recipe for generating paired text-image training data for VLMs to the UI domain by combining ex…2025YJYue Jiang et al.Aalto UniversityVoice User Interface (VUI) DesignHuman-LLM CollaborationIUI
Misty: UI Prototyping Through Interactive Conceptual BlendingUI prototyping often involves iterating and blending elements from examples such as screenshots and sketches, but current tools offer limited support for incorporating these examples. Inspired by the cognitive process of conceptual blending, we introduce a novel UI workflow that allows developers to rapidly incorporat…2025YLYuwen Lu et al.University of Notre DameKnowledge Worker Tools & WorkflowsPrototyping & User TestingCHI
Towards Automated Accessibility Report Generation for Mobile AppsACM DL: https://dl.acm.org/doi/full/10.1145/3674967 Many apps have basic accessibility issues, like missing labels or low contrast. To supplement manual testing, automated tools can help developers and QA testers find basic accessibility issues, but they can be laborious to use or require writing dedicated tests. To…2024ASAmanda Swearngin et al.AppleVoice AccessibilityUniversal & Inclusive DesignPrivacy Perception & Decision-MakingUIST
UIClip: A Data-driven Model for Assessing User Interface DesignUser interface (UI) design is a difficult yet important task for ensuring the usability, accessibility, and aesthetic qualities of applications. In our paper, we develop a machine-learned model, UIClip, for assessing the design quality and visual relevance of a UI given its screenshot and natural language description.…2024JWJason Wu et al.Carnegie Mellon University360° Video & Panoramic ContentHuman-LLM CollaborationExplainable AI (XAI)UIST
Towards Automated Accessibility Report Generation for Mobile AppsMany apps have basic accessibility issues, like missing labels or low contrast. To supplement manual testing, automated tools can help developers and QA testers find basic accessibility issues, but they can be laborious to use or require writing dedicated tests. To motivate our work, we interviewed eight accessibility…2024ASAmanda Swearngin et al.AppleVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Universal & Inclusive DesignUIST
AXNav: Replaying Accessibility Tests from Natural LanguageDevelopers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amongst other development milestones. Recently, Large Language Models (L…2024MTMaryam Taeb et al.Florida State UniversityVoice AccessibilityHuman-LLM CollaborationCHI
Never-ending Learning of User InterfacesMachine learning models have been trained to predict semantic information about user interfaces (UIs) to make apps more accessible, easier to test, and to automate. Currently, most models rely on datasets that are collected and labeled by human crowd-workers, a process that is costly and surprisingly error-prone for c…2023JWJason Wu et al.Carnegie Mellon UniversityHuman-LLM CollaborationAI-Assisted Decision-Making & AutomationAutoML InterfacesUIST
Understanding Screen Relationships from Screenshots of Smartphone ApplicationsAll graphical user interfaces are comprised of one or more screens that may be shown to the user depending on their interactions. Identifying different screens of an app and understanding the type of changes that happen on the screens is a challenging task that can be applied in many areas including automatic app cra…2022SFShirin Feiz et al.Stony Brook UniversityEye Tracking & Gaze InteractionContext-Aware ComputingIUI
Towards Complete Icon Labeling in Mobile ApplicationsAccurately recognizing icon types in mobile applications is integral to many tasks, including accessibility improvement, UI design search, and conversational agents. Existing research focuses on recognizing the most frequent icon types, but these technologies fail when encountering an unrecognized low-frequency icon.…2022JCJieshan Chen et al.Australian National UniversityHuman-LLM CollaborationRecommender System UXCHI
Screen Parsing: Towards Reverse Engineering of UI Models from ScreenshotsAutomated understanding of user interfaces (UIs) from their pixels can improve accessibility, enable task automation, and facilitate interface design without relying on developers to comprehensively provide metadata. A first step is to infer what UI elements exist on a screen, but current approaches are limited in h…2021JWJason Wu et al.Carnegie Mellon UniversityInteractive Data VisualizationPrototyping & User TestingUIST
Screen Recognition: Creating Accessibility Metadata for Mobile Applications from PixelsMany accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sufficient metadata for accessibility features to work as expected. In this paper, we explore inferring acce…2021XZXiaoyi Zhang et al.AppleVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Universal & Inclusive DesignCHI
Swire: Sketch-based User Interface RetrievalSketches and real-world user interface examples are frequently used in multiple stages of the user interface design process. Unfortunately, finding relevant user interface examples, especially in large-scale datasets, is a highly challenging task because user interfaces have aesthetic and functional properties that ar…2019FHForrest Huang et al.University of CaliforniaImmersion & Presence ResearchPrototyping & User TestingCHI