All

Papers

Browse and search HCI research papers from All

Active Filters

Author: 6492

12 results

Improving User Interface Generation Models from Designer Feedback

Despite being trained on vast amounts of data, most LLMs are unable to reliably generate well-designed UIs. Designer feedback is essential to improving performance on UI generation; however, we find that existing RLHF methods based on ratings or rankings are not well-aligned with with designers' workflows and ignore t…

Jason Wu et al.Carnegie Mellon University

From Interaction to Impact: Towards Safer AI Agent Through Understanding and Evaluating Mobile UI Operation Impacts

With advances in generative AI, there is increasing work towards creating autonomous agents that can manage daily tasks by operating user interfaces (UIs). While prior research has studied the mechanics of how AI agents might navigate UIs and understand UI structure, the effects of agents and their autonomous actions—…

Zhuohao (Jerry) Zhang et al.University Of Washington

ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations

Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data. In this paper, we adapt a recipe for generating paired text-image training data for VLMs to the UI domain by combining ex…

Yue Jiang et al.Aalto University

Misty: UI Prototyping Through Interactive Conceptual Blending

UI prototyping often involves iterating and blending elements from examples such as screenshots and sketches, but current tools offer limited support for incorporating these examples. Inspired by the cognitive process of conceptual blending, we introduce a novel UI workflow that allows developers to rapidly incorporat…

Yuwen Lu et al.University of Notre Dame

AdRecommended

Learn AI Coding at CodeNow

Structured lessons, hands-on projects, and continuous updates for people bringing AI into real development work.

Explore Now

Towards Automated Accessibility Report Generation for Mobile Apps

ACM DL: https://dl.acm.org/doi/full/10.1145/3674967 Many apps have basic accessibility issues, like missing labels or low contrast. To supplement manual testing, automated tools can help developers and QA testers find basic accessibility issues, but they can be laborious to use or require writing dedicated tests. To…

Amanda Swearngin et al.Apple

UIClip: A Data-driven Model for Assessing User Interface Design

User interface (UI) design is a difficult yet important task for ensuring the usability, accessibility, and aesthetic qualities of applications. In our paper, we develop a machine-learned model, UIClip, for assessing the design quality and visual relevance of a UI given its screenshot and natural language description.…

Jason Wu et al.Carnegie Mellon University

Towards Automated Accessibility Report Generation for Mobile Apps

Many apps have basic accessibility issues, like missing labels or low contrast. To supplement manual testing, automated tools can help developers and QA testers find basic accessibility issues, but they can be laborious to use or require writing dedicated tests. To motivate our work, we interviewed eight accessibility…

Amanda Swearngin et al.Apple

AXNav: Replaying Accessibility Tests from Natural Language

Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amongst other development milestones. Recently, Large Language Models (L…

Maryam Taeb et al.Florida State University

Never-ending Learning of User Interfaces

Machine learning models have been trained to predict semantic information about user interfaces (UIs) to make apps more accessible, easier to test, and to automate. Currently, most models rely on datasets that are collected and labeled by human crowd-workers, a process that is costly and surprisingly error-prone for c…

Jason Wu et al.Carnegie Mellon University

Understanding Screen Relationships from Screenshots of Smartphone Applications

All graphical user interfaces are comprised of one or more screens that may be shown to the user depending on their interactions. Identifying different screens of an app and understanding the type of changes that happen on the screens is a challenging task that can be applied in many areas including automatic app cra…

Shirin Feiz et al.Stony Brook University

Towards Complete Icon Labeling in Mobile Applications

Accurately recognizing icon types in mobile applications is integral to many tasks, including accessibility improvement, UI design search, and conversational agents. Existing research focuses on recognizing the most frequent icon types, but these technologies fail when encountering an unrecognized low-frequency icon.…

Jieshan Chen et al.Australian National University

Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sufficient metadata for accessibility features to work as expected. In this paper, we explore inferring acce…

Xiaoyi Zhang et al.Apple

Paper Title	Authors	Research Topics	Paper Database	Year
Improving User Interface Generation Models from Designer Feedback Despite being trained on vast amounts of data, most LLMs are unable to reliably generate well-designed UIs. Designer feedback is essential to improving performance on UI generation; however, we find that existing RLHF methods based on ratings or rankings are not well-aligned with with designers' workflows and ignore t…	JW Jason Wu et al.Carnegie Mellon University
From Interaction to Impact: Towards Safer AI Agent Through Understanding and Evaluating Mobile UI Operation Impacts With advances in generative AI, there is increasing work towards creating autonomous agents that can manage daily tasks by operating user interfaces (UIs). While prior research has studied the mechanics of how AI agents might navigate UIs and understand UI structure, the effects of agents and their autonomous actions—…	ZZ Zhuohao (Jerry) Zhang et al.University Of Washington
ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data. In this paper, we adapt a recipe for generating paired text-image training data for VLMs to the UI domain by combining ex…	YJ Yue Jiang et al.Aalto University
Misty: UI Prototyping Through Interactive Conceptual Blending UI prototyping often involves iterating and blending elements from examples such as screenshots and sketches, but current tools offer limited support for incorporating these examples. Inspired by the cognitive process of conceptual blending, we introduce a novel UI workflow that allows developers to rapidly incorporat…	YL Yuwen Lu et al.University of Notre Dame
AdRecommended Learn AI Coding at CodeNow Structured lessons, hands-on projects, and continuous updates for people bringing AI into real development work. Explore Now
Towards Automated Accessibility Report Generation for Mobile Apps ACM DL: https://dl.acm.org/doi/full/10.1145/3674967 Many apps have basic accessibility issues, like missing labels or low contrast. To supplement manual testing, automated tools can help developers and QA testers find basic accessibility issues, but they can be laborious to use or require writing dedicated tests. To…	AS Amanda Swearngin et al.Apple
UIClip: A Data-driven Model for Assessing User Interface Design User interface (UI) design is a difficult yet important task for ensuring the usability, accessibility, and aesthetic qualities of applications. In our paper, we develop a machine-learned model, UIClip, for assessing the design quality and visual relevance of a UI given its screenshot and natural language description.…	JW Jason Wu et al.Carnegie Mellon University
Towards Automated Accessibility Report Generation for Mobile Apps Many apps have basic accessibility issues, like missing labels or low contrast. To supplement manual testing, automated tools can help developers and QA testers find basic accessibility issues, but they can be laborious to use or require writing dedicated tests. To motivate our work, we interviewed eight accessibility…	AS Amanda Swearngin et al.Apple
AXNav: Replaying Accessibility Tests from Natural Language Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amongst other development milestones. Recently, Large Language Models (L…	MT Maryam Taeb et al.Florida State University
Never-ending Learning of User Interfaces Machine learning models have been trained to predict semantic information about user interfaces (UIs) to make apps more accessible, easier to test, and to automate. Currently, most models rely on datasets that are collected and labeled by human crowd-workers, a process that is costly and surprisingly error-prone for c…	JW Jason Wu et al.Carnegie Mellon University
Understanding Screen Relationships from Screenshots of Smartphone Applications All graphical user interfaces are comprised of one or more screens that may be shown to the user depending on their interactions. Identifying different screens of an app and understanding the type of changes that happen on the screens is a challenging task that can be applied in many areas including automatic app cra…	SF Shirin Feiz et al.Stony Brook University
Towards Complete Icon Labeling in Mobile Applications Accurately recognizing icon types in mobile applications is integral to many tasks, including accessibility improvement, UI design search, and conversational agents. Existing research focuses on recognizing the most frequent icon types, but these technologies fail when encountering an unrecognized low-frequency icon.…	JC Jieshan Chen et al.Australian National University
emoji_events Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels Many accessibility features available on mobile platforms require applications (apps) to provide complete and accurate metadata describing user interface (UI) components. Unfortunately, many apps do not provide sufficient metadata for accessibility features to work as expected. In this paper, we explore inferring acce…	XZ Xiaoyi Zhang et al.Apple