Automating UI Optimization through Multi-Agentic Reasoning

Paper Title

Automating UI Optimization through Multi-Agentic Reasoning

Publication Info

• Topic area: Adaptive user interface optimization in mixed reality using multi-agent systems.
• Keywords: UI optimization, mixed reality, vision-language models, multi-objective optimization, Pareto front, ambiguity detection, user preferences, adaptive interfaces, validation module, human-computer interaction.

Background and Problem

• Problem / challenge: Current optimization methods for user interfaces rely heavily on manual configuration by designers and users, making them labor-intensive and unable to dynamically adapt to individual user preferences or task-specific contexts.
• Significance: Automating the optimization process can reduce user effort, improve alignment with individual preferences, and enable dynamic adaptation of UIs, particularly in complex environments like mixed reality.
• Motivation and related work: Prior work has explored adaptive UIs in mixed reality, focusing on ergonomic and environment-based optimization. However, these methods assume fixed objective functions and parameters, limiting their ability to tailor UIs to individual user needs. Multi-objective optimization techniques generate Pareto-optimal solutions but require users to manually evaluate and select designs. Recent advances in vision-language models (VLMs) and large language models (LLMs) have demonstrated their potential for reasoning about user preferences and automating complex tasks, providing a foundation for this work.

Solution

• Proposed approach: AutoOptimization, a multi-agent framework that uses vision-language models to automate the optimization of user interfaces based on verbal user instructions.
• Novelty:
  1. Automates the setup and decision-making phases of UI optimization using VLMs as proxy designers.
  2. Integrates ambiguity detection, configuration, optimization, and validation into a single pipeline to streamline UI adaptation.
  3. Dynamically formulates multi-objective optimization problems based on user instructions and selects Pareto-optimal solutions.
  4. Reduces manual effort while improving alignment with user preferences compared to baseline methods.
• Procedure and key techniques:
  • Ambiguity Detection: Identifies unclear aspects of user instructions and iteratively prompts for clarification until sufficient information is gathered.
  • Configuration: Translates clarified instructions into a multi-objective optimization problem by selecting relevant objectives, widgets, and parameters.
  • Optimization: Solves the optimization problem using NSGA-III to generate Pareto-optimal layout candidates.
  • Validation: Evaluates candidate layouts against user instructions and selects the most suitable design using VLM reasoning.

Results

• Concrete findings:
  • Ambiguity detection achieved 91.26% accuracy in leave-one-user-out cross-validation and 92.93% accuracy in leave-one-scenario-out cross-validation.
  • Validation module showed a 100% overlap with human participants in selecting layouts that best aligned with user instructions.
  • User study results indicated fewer adjustments and shorter adjustment distances with AutoOptimization compared to ParetoAdapt and manual placement methods.
  • NASA-TLX scores revealed lower physical demand and overall workload with AutoOptimization compared to baseline methods, despite higher mental demand.
• Advantage over baselines:
  • AutoOptimization reduced manual effort and generated layouts better aligned with user preferences than ParetoAdapt.
  • Achieved comparable user satisfaction to manual placement while significantly reducing physical effort and adjustment time.
• Experiments / evaluation:
  • Conducted ambiguity detection tests, layout comparison surveys, and a user study with 12 participants across three scenarios (living room, office, airplane).
  • Metrics included adjustment distance, number of adjustments, NASA-TLX scores, and user rankings of generated layouts.
• Limitations and future work:
  • Limited to objective-space instructions; does not support direct design-space commands.
  • Requires manual refinement post-optimization for low-level adjustments.
  • Clarification questions may unintentionally shape user preferences.
  • Assumes user-desired outcomes lie on the Pareto front, which may not always hold true.
  • Future directions include integrating direct manipulation, hybrid optimization approaches, and personalized few-shot learning.

Summary

AutoOptimization is a novel framework that automates UI optimization using multi-agent reasoning and vision-language models. By dynamically interpreting user instructions, configuring optimization problems, and validating results, it reduces manual effort while improving alignment with individual preferences. Evaluation in mixed reality demonstrated its effectiveness in ambiguity detection, layout validation, and user satisfaction, outperforming baseline methods. While limitations remain in handling design-space instructions and post-optimization adjustments, the framework represents a significant step toward personalized, adaptive UIs. Future work will focus on expanding objective functions, integrating direct manipulation, and refining user preference elicitation.