DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfaces

Paper Title

DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfaces

Publication Info

• Topic area: Human-agent collaboration in task-oriented interface generation
• Keywords: Human-agent co-generation, bidirectional context loop, task-oriented interfaces, large language models, user interface generation, human-computer interaction, mixed-initiative systems, end-user development, task automation, interface usability

Background and Problem

• Problem / challenge: Existing task automation systems, including those powered by LLMs, struggle with complex, multi-step workflows due to gaps between user intent and agent actions. Current paradigms are either overly agent-centric or require users to adapt to rigid workflows, limiting fluid human-agent collaboration.
• Significance: Addressing these challenges can improve task efficiency, usability, and user satisfaction in real-world, multi-step tasks, making AI systems more accessible and effective for non-expert users.
• Motivation and related work: Prior work in task automation and UI generation has evolved from direct manipulation to full automation and human-in-the-loop approaches. However, these paradigms fail to fully integrate user intent with agent actions dynamically. This paper builds on concepts like end-user development and mixed-initiative systems to propose a new collaborative paradigm.

Solution

• Proposed approach: DuetUI, a system embodying the human-agent co-generation paradigm, operationalized through a bidirectional context loop. The agent decomposes tasks into interface scaffolds, while user interactions guide subsequent agent actions.
• Novelty:
  1. Introduction of the human-agent co-generation paradigm with a bidirectional context loop.
  2. Development of DuetUI, a system that integrates task decomposition and user-driven interface manipulation.
  3. Empirical evaluation demonstrating improved task efficiency, usability, and collaboration.
• Procedure and key techniques:
  • DuetUI structures interaction into six stages: Define, Empathize, Plan, Explore, Refine, and Duet.
  • Features include Staged Co-Generation, Tangible Agency (manipulable UI components), Task-Interface Duality (semantic alignment between tasks and UI), and Bidirectional Action History (shared log of user and agent actions).
  • A three-layer architecture (Core, Context, Agent) operationalizes the bidirectional context loop.

Results

• Concrete findings:
  • DuetUI achieved a weighted F1 score of 0.508 in automated evaluations, outperforming baselines.
  • User study results showed higher usability (SUS: 73.65 vs. 63.47), task satisfaction (3.7 vs. 3.25), interface satisfaction (3.96 vs. 3.54), and AI satisfaction (3.84 vs. 3.44) compared to the baseline.
  • Participants interacted more frequently with DuetUI's interface widgets (6.6 interactions/task vs. 1.2 for the baseline).
• Advantage over baselines:
  • DuetUI outperformed baseline systems in task efficiency, usability, and alignment with user intent.
  • Enabled iterative, dynamic collaboration, reducing user reliance on textual prompts.
• Experiments / evaluation:
  • Conducted a technical ablation study comparing DuetUI with and without the bidirectional loop.
  • User study with 24 participants evaluated DuetUI against a baseline system (Stitch) across 10 real-world tasks.
  • Metrics included SUS, NASA-TLX, and satisfaction ratings, alongside qualitative feedback.
• Limitations and future work:
  • Reduced precision in automated evaluation due to additional generated elements.
  • Limited generalizability due to a small sample size and reliance on simulated external services.
  • Future work includes adaptive control mechanisms, richer contextual understanding, and real-world deployment with persistent user profiles.

Summary

DuetUI introduces a novel human-agent co-generation paradigm, enabling dynamic collaboration through a bidirectional context loop. The system demonstrated significant improvements in task efficiency, usability, and user satisfaction compared to baseline approaches. By integrating explicit and implicit user inputs, DuetUI bridges the gap between user intent and agent actions, fostering a fluid partnership. While the approach shows promise, further research is needed to address limitations in precision, scalability, and real-world applicability.