Yue Jiang

schoolAalto Universitybar_chart#1842 Global Rank

Total Papers

HCI Rank

#1842

Research Areas

Years Active

Recent Publications

SeekUI: Predicting Visual Search Behavior on Graphical User Interfaces with a Reward-Augmented Vision Language Model

Visual search is key to understanding and improving interaction with graphical user interfaces (GUIs), yet predicting scanpaths on real GUIs remains an open challenge. Unlike free-viewing, visual search is goal-driven and shaped by both linguistic and visual features of the GUI. State-of-the-art models of visual searc…

IUI '251st AuthorVoice User Interface (VUI) Design +1

ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations

Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data. In this paper, we adapt a recipe for generating paired text-image training data for VLMs to the UI domain by combining ex…

UIST '241st AuthorEye Tracking & Gaze Interaction +2

EyeFormer: Predicting Personalized Scanpaths with Transformer-Guided Reinforcement Learning

From a visual-perception perspective, modern graphical user interfaces (GUIs) comprise a complex graphics-rich two-dimensional visuospatial arrangement of text, images, and interactive objects such as buttons and menus. While existing models can accurately predict regions and objects that are likely to attract attenti…

CHI '245th+ AuthorVoice Accessibility +1

AXNav: Replaying Accessibility Tests from Natural Language

Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amongst other development milestones. Recently, Large Language Models (L…

CHI '241st Author360° Video & Panoramic Content +1

Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces

Present-day graphical user interfaces (GUIs) exhibit diverse arrangements of text, graphics, and interactive elements such as buttons and menus, but representations of GUIs have not kept up. They do not encapsulate both semantic and visuo-spatial relationships among elements. To seize machine learning's potential for…

CHI '231st AuthorEye Tracking & Gaze Interaction +1

UEyes: Understanding Visual Saliency across User Interface Types

While user interfaces (UIs) display elements such as images and text in a grid-based layout, UI types differ significantly in the number of elements and how they are displayed. For example, webpage designs rely heavily on images and text, whereas desktop UIs tend to feature numerous small images. To examine how such d…

Quick Actions

Share this page

ios_share

X in mail

https://hci.top/en/authors/9992?source=all