The GenUI Study: Exploring the Design of Generative UI Tools to Support UX Practitioners and BeyondAI can now generate high-fidelity UI mock-up screens from a high-level textual description, promising to support UX practitioners' work. However, it remains unclear how UX practitioners would adopt such Generative UI (GenUI) models in a way that is integral and beneficial to their work. To answer this question, we con…2025XCXiang 'Anthony' Chen et al.University of California, Los AngelesGenerative AI (Text, Image, Music, Video)Human-LLM CollaborationAI-Assisted Creative WritingDIS
UICrit: Enhancing Automated Design Evaluation with a UI Critique DatasetAutomated UI evaluation can be beneficial for the design process; for example, to compare different UI designs, or conduct automated heuristic evaluation. LLM-based UI evaluation, in particular, holds the promise of generalizability to a wide variety of UI types and evaluation tasks. However, current LLM-based techniq…2024PDPeitong Duan et al.University of California - BerkeleyHuman-LLM CollaborationExplainable AI (XAI)UIST
Automatic Macro Mining from Interaction Traces at ScaleMacros are building block tasks of our everyday smartphone activity (e.g., "login", or "booking a flight"). Effectively extracting macros is important for understanding mobile interaction and enabling task automation. These macros are however difficult to extract at scale as they can be comprised of multiple steps yet…2024FHForrest Huang et al.GoogleHuman-LLM CollaborationAI-Assisted Decision-Making & AutomationCHI
Generating Automatic Feedback on UI Mockups with Large Language ModelsFeedback on user interface (UI) mockups is crucial in design. However, human feedback is not always readily available. We explore the potential of using large language models for automatic feedback. Specifically, we focus on \changes{applying GPT-4 to automate heuristic evaluation}, which currently entails a human exp…2024PDPeitong Duan et al.University of California - BerkeleyHuman-LLM CollaborationPrototyping & User TestingCHI
Enabling Conversational Interaction with Mobile UI using Large Language ModelsConversational agents show the promise to allow users to interact with mobile devices using language. However, to perform diverse UI tasks with natural language, developers typically need to create separate datasets and models for each specific task, which is expensive and effort-consuming. Recently, pre-trained large…2023BWBryan Wang et al.University of TorontoVoice User Interface (VUI) DesignHuman-LLM CollaborationCHI
Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency AnalysisUI designers often correct false affordances and improve the discoverability of features when users have trouble determining if elements are tappable. We contribute a novel system that models the perceived tappability of mobile UI elements with a vision-based deep neural network and helps provide design insights with…2022ESEldon Schoop et al.University of CaliforniaExplainable AI (XAI)AI-Assisted Decision-Making & AutomationCHI
Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at ScaleThe layout of a mobile screen is a critical data source for UI design research and semantic understanding of the screen. However, UI layouts in existing datasets are often noisy, have mismatches with their visual representation, or consists of generic or app-specific types that are difficult to analyze and model. In t…2022GLGang Li et al.GooglePrototyping & User TestingComputational Methods in HCICHI
Screen2Words: Automatic Mobile UI Summarization with Multimodal LearningMobile User Interface Summarization generates succinct language descriptions of mobile screens for conveying important contents and functionalities of the screen, which can be useful for many language-based application scenarios. We present Screen2Words, a novel screen summarization approach that automatically encapsu…2021BWBryan Wang et al.University of TorontoVoice User Interface (VUI) DesignHuman-LLM CollaborationUIST
HelpViz: Automatic Generation of Contextual Visual Mobile Tutorials from Text-Based InstructionsWe present HelpViz, a tool for generating contextual visual mobile tutorials from text-based instructions that are abundant on the web. HelpViz transforms text instructions to graphical tutorials in batch, by extracting a sequence of actions from each text instruction through an instruction parsing model, and executin…2021MZMingyuan Zhong et al.University Of WashingtonInteractive Data VisualizationData StorytellingUIST
Spacewalker: Rapid UI Design Exploration Using Lightweight Markup Enhancement and Crowd Genetic ProgrammingUser interface design is a complex task that involves designers examining a wide range of options. We present Spacewalker, a tool that allows designers to rapidly search a large design space for an optimal web UI with integrated support. Designers first annotate each attribute they want to explore in a typical HTML pa…2021MZMingyuan Zhong et al.University Of WashingtonCrowdsourcing Task Design & Quality ControlPrototyping & User TestingCHI
TapNet: The Design, Training, Implementation, and Applications of a Multi-Task Learning CNN for Off-Screen Mobile InputTo make off-screen interaction without specialized hardware practical, we investigate using deep learning methods to process the common built-in IMU sensor (accelerometers and gyroscopes) on mobile phones into a useful set of one-handed interaction events. We present the design, training, implementation and applicat…2021MHMichael Xuelin Huang et al.Saarland Informatics CampusFoot & Wrist InteractionComputational Methods in HCICHI
Modeling Human Visual Search Performance on Realistic Webpages Using Analytical and Deep Learning MethodsModeling visual search not only offers an opportunity to predict the usability of an interface before actually testing it on real users but also advances scientific understanding about human behavior. In this work, we first conduct a set of analyses on a large-scale dataset of visual search tasks on realistic webpages…2020AYArianna Yuan et al.Stanford UniversityExplainable AI (XAI)Visualization Perception & CognitionCHI
Modeling Mobile Interface Tappability Using Crowdsourcing and Deep LearningTapping is an immensely important gesture in mobile touchscreen interfaces, yet people still frequently are required to learn which elements are tappable through trial and error. Predicting human behavior for this everyday gesture can help mobile app designers understand an important aspect of the usability of their a…2019ASAmanda M Swearngin et al.University Of WashingtonPrototyping & User TestingComputational Methods in HCICHI
Predicting Human Performance in Vertical Menu Selection Using Deep LearningPredicting human performance in interaction tasks allows designers or developers to understand the expected performance of a target interface without actually testing it with real users. In this work, we present a deep neural net to model and predict human performance in performing a sequence of UI tasks. In particula…2018YLYang Li et al.GoogleRecommender System UXComputational Methods in HCICHI
Analysis and Modeling of Grid Performance on Touchscreen Mobile DevicesTouchscreen mobile devices can afford rich interaction behaviors but they are complex to model. Scrollable two-dimensional grids are a common user interface on mobile devices that allow users to access a large number of items on a small screen by direct touch. By analyzing touch input and eye gaze of users during grid…2018KPKen Pfeuffer et al.Aarhus UniversityEye Tracking & Gaze InteractionInteractive Data VisualizationCHI
M3 Gesture Menu: Design and Experimental Analyses of Marking Menus for Touchscreen Mobile InteractionDespite their learning advantages in theory, marking menus have faced adoption challenges in practice, even on today's touchscreen-based mobile devices. We address these challenges by designing, implementing, and evaluating multiple versions of M3 Gesture Menu (M3), a reimagination of marking menus targeted at mobile…2018JZJingjie Zheng et al.University of WaterlooHand Gesture RecognitionCHI
Doppio: Tracking UI Flows and Code Changes for App DevelopmentDeveloping interactive systems often involves a large set of callback functions for handling user interaction, which makes it challenging to manage UI behaviors, create descriptive documentation, and track code revisions. We developed Doppio, a tool that automatically tracks and visualizes UI flows and their changes b…2018PCPei-Yu (Peggy) Chi et al.GoogleKnowledge Worker Tools & WorkflowsComputational Methods in HCICHI