Jeffrey Nichols

schoolGooglebar_chart#307 Global Rank

Total Papers

HCI Rank

#307

Research Areas

Years Active

Recent Publications

Mapping the Design Space of User Experience for Computer Use Agents

Large language model (LLM)-based computer use agents execute user commands by interacting with available UI elements, but little is known about how users want to interact with these agents or what design factors matter for their user experience (UX). We conducted a two-phase study to map the UX design space for comput…

IUI '26CorrespondingHuman-LLM Collaboration +2

Athena: Intermediate Representations for Iterative Scaffolded App Generation with an LLM

It is challenging to generate the code for a complete user interface using a Large Language Model (LLM). User interfaces are complex and their implementations often consist of multiple, inter-related files that together specify the contents of each screen, the navigation flows between the screens, and the data model u…

CHI '263rd AuthorHuman-LLM Collaboration +2

The Way We Notice, That’s What Really Matters: Instantiating UI Components with Distinguishing Variations

Front-end developers author UI components to be broadly reusable by parameterizing visual and behavioral properties. While flexible, this makes instantiation harder, as developers must reason about numerous property values and interactions. In practice, they must explore the component’s large design space and provide…

CHI '265th+ AuthorHuman-LLM Collaboration +2

Improving User Interface Generation Models from Designer Feedback

Despite being trained on vast amounts of data, most LLMs are unable to reliably generate well-designed UIs. Designer feedback is essential to improving performance on UI generation; however, we find that existing RLHF methods based on ratings or rankings are not well-aligned with with designers' workflows and ignore t…

UIST '254th AuthorHuman-LLM Collaboration +1

SQUIRE: Interactive UI Authoring via Slot QUery Intermediate REpresentations

Frontend developers create UI prototypes to evaluate alternatives, which is a time-consuming process of repeated iteration and refinement. Generative AI code assistants enable rapid prototyping simply by prompting through a chat interface rather than writing code. However, while this interaction gives developers flexi…

IUI '253rd AuthorGenerative AI (Text, Image, Music, Video) +1

From Interaction to Impact: Towards Safer AI Agent Through Understanding and Evaluating Mobile UI Operation Impacts

With advances in generative AI, there is increasing work towards creating autonomous agents that can manage daily tasks by operating user interfaces (UIs). While prior research has studied the mechanics of how AI agents might navigate UIs and understand UI structure, the effects of agents and their autonomous actions—…

IUI '25CorrespondingVoice User Interface (VUI) Design +1

ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations

Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data. In this paper, we adapt a recipe for generating paired text-image training data for VLMs to the UI domain by combining ex…

CHI '254th AuthorKnowledge Worker Tools & Workflows +1

Misty: UI Prototyping Through Interactive Conceptual Blending

UI prototyping often involves iterating and blending elements from examples such as screenshots and sketches, but current tools offer limited support for incorporating these examples. Inspired by the cognitive process of conceptual blending, we introduce a novel UI workflow that allows developers to rapidly incorporat…

UIST '24CorrespondingVoice Accessibility +2

Towards Automated Accessibility Report Generation for Mobile Apps

ACM DL: https://dl.acm.org/doi/full/10.1145/3674967 Many apps have basic accessibility issues, like missing labels or low contrast. To supplement manual testing, automated tools can help developers and QA testers find basic accessibility issues, but they can be laborious to use or require writing dedicated tests. To…

UIST '24Corresponding360° Video & Panoramic Content +2

UIClip: A Data-driven Model for Assessing User Interface Design

User interface (UI) design is a difficult yet important task for ensuring the usability, accessibility, and aesthetic qualities of applications. In our paper, we develop a machine-learned model, UIClip, for assessing the design quality and visual relevance of a UI given its screenshot and natural language description.…

View all 18 papers →

Quick Actions

Share this page

ios_share

X in mail

https://hci.top/en/authors/2601?source=all