Amanda Swearngin

schoolApplebar_chart#671 Global Rank

Total Papers

HCI Rank

#671

Research Areas

Years Active

Recent Publications

Improving User Interface Generation Models from Designer Feedback

Despite being trained on vast amounts of data, most LLMs are unable to reliably generate well-designed UIs. Designer feedback is essential to improving performance on UI generation; however, we find that existing RLHF methods based on ratings or rankings are not well-aligned with with designers' workflows and ignore t…

IUI '25CorrespondingGenerative AI (Text, Image, Music, Video) +1

From Interaction to Impact: Towards Safer AI Agent Through Understanding and Evaluating Mobile UI Operation Impacts

With advances in generative AI, there is increasing work towards creating autonomous agents that can manage daily tasks by operating user interfaces (UIs). While prior research has studied the mechanics of how AI agents might navigate UIs and understand UI structure, the effects of agents and their autonomous actions—…

IUI '253rd AuthorVoice User Interface (VUI) Design +1

ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations

Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data. In this paper, we adapt a recipe for generating paired text-image training data for VLMs to the UI domain by combining ex…

CHI '253rd AuthorKnowledge Worker Tools & Workflows +1

Misty: UI Prototyping Through Interactive Conceptual Blending

UI prototyping often involves iterating and blending elements from examples such as screenshots and sketches, but current tools offer limited support for incorporating these examples. Inspired by the cognitive process of conceptual blending, we introduce a novel UI workflow that allows developers to rapidly incorporat…

UIST '241st AuthorVoice Accessibility +2

Towards Automated Accessibility Report Generation for Mobile Apps

ACM DL: https://dl.acm.org/doi/full/10.1145/3674967 Many apps have basic accessibility issues, like missing labels or low contrast. To supplement manual testing, automated tools can help developers and QA testers find basic accessibility issues, but they can be laborious to use or require writing dedicated tests. To…

UIST '244th Author360° Video & Panoramic Content +2

UIClip: A Data-driven Model for Assessing User Interface Design

User interface (UI) design is a difficult yet important task for ensuring the usability, accessibility, and aesthetic qualities of applications. In our paper, we develop a machine-learned model, UIClip, for assessing the design quality and visual relevance of a UI given its screenshot and natural language description.…

UIST '241st AuthorVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille) +1

Towards Automated Accessibility Report Generation for Mobile Apps

Many apps have basic accessibility issues, like missing labels or low contrast. To supplement manual testing, automated tools can help developers and QA testers find basic accessibility issues, but they can be laborious to use or require writing dedicated tests. To motivate our work, we interviewed eight accessibility…

CHI '242nd AuthorVoice Accessibility +1

AXNav: Replaying Accessibility Tests from Natural Language

Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amongst other development milestones. Recently, Large Language Models (L…

UIST '234th AuthorHuman-LLM Collaboration +2

Never-ending Learning of User Interfaces

Machine learning models have been trained to predict semantic information about user interfaces (UIs) to make apps more accessible, easier to test, and to automate. Currently, most models rely on datasets that are collected and labeled by human crowd-workers, a process that is costly and surprisingly error-prone for c…

IUI '224th AuthorEye Tracking & Gaze Interaction +1

Understanding Screen Relationships from Screenshots of Smartphone Applications

All graphical user interfaces are comprised of one or more screens that may be shown to the user depending on their interactions. Identifying different screens of an app and understanding the type of changes that happen on the screens is a challenging task that can be applied in many areas including automatic app cra…

View all 12 papers →

Quick Actions

Share this page

ios_share

X in mail

https://hci.top/en/authors/6492?source=all