Interaction-Augmented Instruction: Modeling the Synergy of Prompts and Interactions in Human-GenAI Collaboration

Paper Title

Interaction-Augmented Instruction: Modeling the Synergy of Prompts and Interactions in Human-GenAI Collaboration

Publication Info

• Topic area: Enhancing human–GenAI communication through the integration of text prompts and GUI interactions.
• Keywords: Generative AI, human–AI collaboration, interaction design, augmented instruction, GUI interactions, text prompts, interface paradigms, HCI, multimodal communication, design frameworks.

Background and Problem

• Problem / challenge: Current GenAI systems rely heavily on text prompts, which are often ambiguous and insufficient for expressing fine-grained or referential intent. Existing interaction paradigms lack a unified model to systematically capture and compare designs that combine text prompts with GUI interactions.
• Significance: Addressing this gap is crucial for improving the precision, transparency, and usability of GenAI systems, enabling more effective human–AI collaboration across diverse domains such as design, education, and data analysis.
• Motivation and related work: Prior research has explored taxonomies of interaction paradigms and principles for human–GenAI communication but remains fragmented, descriptive, or limited in generative power. This paper builds on these efforts by introducing a formal model to unify and extend the design space.

Solution

• Proposed approach: The Interaction-Augmented Instruction (IAI) model, a formal entity–relation graph that captures how text prompts and GUI interactions synergistically form augmented instructions for GenAI systems.
• Novelty:
  1. Introduction of the IAI model with six core entities (Human, Interaction, Text Prompt, Augmented Instruction, Artifact, Generative AI) and their relations.
  2. Distillation of 12 recurring atomic paradigms for interaction-augmented instruction, categorized by interaction timing and user resources.
  3. Demonstration of the model’s descriptive, discriminative, and generative power through usage scenarios and design applications.
• Procedure and key techniques:
  1. Develop the IAI model through iterative and deductive analysis, defining entities and relations to capture human–GenAI workflows.
  2. Annotate 66 GenAI system interfaces to identify and categorize 12 atomic interaction paradigms.
  3. Apply the model to guide the design, refinement, and innovation of GenAI interfaces across four usage scenarios.

Results

• Concrete findings:
  • The IAI model successfully captures and differentiates diverse GenAI interaction workflows.
  • Twelve atomic paradigms were identified, spanning pre- and post-invocation interactions and prompt-only vs. artifact-grounded scenarios.
  • Usage scenarios demonstrate the model’s applicability in extending, refining, and generating new interface designs.
• Advantage over baselines:
  • Greater descriptive and discriminative power compared to prior taxonomies by explicitly modeling the interplay of text prompts, interactions, and artifacts.
  • Generative capability to inspire new paradigms and interface designs, addressing gaps in existing frameworks.
• Experiments / evaluation:
  • Analysis of 66 GenAI tools to validate the model’s descriptive and discriminative power.
  • Four illustrative usage scenarios to demonstrate practical applications of the model and paradigms.
• Limitations and future work:
  • Focus on single-user, single-agent interactions; does not address multi-agent workflows.
  • Derived paradigms are based on a limited corpus of 66 tools, which may not fully capture the evolving GenAI ecosystem.
  • Future work includes extending the model to finer granularity, expanding the paradigm collection, and conducting empirical evaluations.

Summary

This paper introduces the Interaction-Augmented Instruction (IAI) model, a formal framework for integrating text prompts and GUI interactions in human–GenAI communication. By defining six core entities and their relations, the model enables systematic characterization, comparison, and innovation in interface design. Twelve atomic paradigms were identified, providing reusable abstractions for diverse design scenarios. Four usage scenarios demonstrate the model’s generative power in refining existing tools and creating new interaction paradigms. The IAI model offers a robust foundation for advancing human–GenAI collaboration, with future work aimed at extending its scope and validating its practical impact.