The Behavioral Fabric of LLM-Powered GUI Agents: Human Values and Interaction Outcomes
Authors
Large Language Model (LLM)-powered web GUI agents are increasingly automating everyday online tasks. Despite their popularity, little is known about how users' preferences and values impact agents' reasoning and behavior. In this work, we investigate how both explicit and implicit user preferences, as well as the underlying user values, influence agent decision-making and action trajectories. We built a controlled testbed of 14 common interactive web tasks, spanning shopping, travel, dining, and housing, each replicated from real websites and integrated with a low-fidelity LLM-based recommender system. We injected 12 human preferences and values as personas into four state-of-the-art agents and systematically analyzed their task behaviors. Our results show that preference and value-infused prompts consistently guided agents toward outcomes that reflected these preferences and values. While the absence of user preference or value guidance led agents to exhibit a strong efficiency bias and employ shortest-path strategies, their presence steered agents' behavior trajectories through the greater use of corresponding filters and interactive web features. Despite their influence, dominant interface cues, such as discounts and advertisements, frequently overrode these effects, shortening the agents' action trajectories and inducing rationalizations that masked rather than reflected value-consistent reasoning. The contributions of this paper are twofold: (1) an open-source testbed for studying the influence of values in agent behaviors, and (2) an empirical investigation of how user preferences and values shape web agent behaviors.
Research Questions / Practical Problems
Question signals indexed for this paper.
- 86%
Whose Code Is It? How AI Autonomy Reshapes Ownership, Responsibility, and Disclosure in AI-Assisted Programming
IUI '26· Human-LLM Collaboration +2
- 71%
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
CHI '24· Human-LLM Collaboration +2
- 71%
"If the Machine Is As Good As Me, Then What Use Am I?" – How the Use of ChatGPT Changes Young Professionals' Perception of Productivity and Accomplishment
CHI '24· Human-LLM Collaboration +1
- 71%
"It would work for me too": How Online Communities Shape Software Developers’ Trust in AI-Powered Code Generation Tools
IUI '25· Human-LLM Collaboration +1
- 71%
Generative Trigger-Action Programming with Ply
UIST '25· Human-LLM Collaboration +1
- 67%
ImpReSS: Designing and Evaluating a Lightweight Implicit Recommender System in Conversational Support Agents
IUI '26· Human-LLM Collaboration +4
- 63%
Planning for Natural Language Failures with the AI Playbook
CHI '21· Human-LLM Collaboration +2
- 63%
Adapting User Interfaces with Model-based Reinforcement Learning
CHI '21· Human-LLM Collaboration +2
- 63%
Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions
CHI '24· Human-LLM Collaboration +2
- 63%
Understanding Socio-technical Factors Configuring AI Non-Use in UX Work Practices
CHI '25· Human-LLM Collaboration +2
Based on Jaccard similarity of research subtopics & professions (≥60%)