From Operation to Cognition: Automatic Modeling Cognitive Dependencies from User Demonstrations for GUI Task Automation

Research Background and Issues

• Problems or Challenges Identified by the Authors: Traditional Programming by Demonstration (PBD) systems primarily rely on recording and replaying user actions within graphical user interfaces (GUIs) to achieve task automation, while largely neglecting the cognitive processes underlying these actions. This approach makes it difficult for task programs to generalize across different contexts, such as collecting and summarizing introductory content from multiple websites based on varying search keywords.

• Importance of the Problem: Current PBD systems are limited in their ability to capture cognitive dependencies during user operations. These cognitive dependencies describe the reasoning and decision-making processes users employ when selecting operation parameters based on information obtained from previous actions. When tasks involve multiple interdependent parameters, this omission significantly reduces the effectiveness of automation.

• Research Motivation and Related Work: Existing research on PBD has primarily focused on constructing task models through symbolic reasoning or enhancing the expressiveness of natural language task descriptions using large language models (LLMs). However, few methods explicitly model cognitive dependencies between operations. To address this gap, this paper introduces a new system, TaskMind, aiming to bridge the gap between operations and cognition, thereby improving the generalization capabilities of task programs across diverse contexts.

Solution

• Proposed Solution: The authors propose TaskMind, an intelligent automation system based on PBD. Its core innovation lies in the introduction of a "task graph," which explicitly models the sequence of user operations and their cognitive dependencies, enabling the system to understand user intentions and dynamically generate task parameters for new contexts.

• Innovations:

  1. Transforming implicit cognitive dependencies into user-comprehensible task graphs.
  2. Introducing an operation-cognition knowledge base to systematically classify types of cognitive dependencies.
  3. Leveraging LLMs to automatically extract cognitive dependencies from user demonstrations and implementing an interactive feedback module for continuous evolution of the knowledge base.

• Implementation Steps:

  1. Task Capture Phase: Users complete tasks through demonstrations, and TaskMind records all GUI operations, keyboard, and mouse events.
  2. Task Analysis Phase:
     • A UI extraction module processes low-level operations into semantically rich operation sets.
     • A dependency extraction module (including structural and semantic deep extractors) utilizes LLMs to capture cognitive dependencies between operations.
  3. Task Customization and Execution Phase: Using the generated task graph, users verify and modify necessary dependencies, define new task objectives, and execute the task.
  4. Reflection and Knowledge Evolution: If users modify the task graph or add new dependency descriptions, the system absorbs these changes through a reflection module and updates the operation-cognition knowledge base, enhancing the model's understanding of future tasks.

• Key Technologies Used:

  • Large Language Models (LLMs) for dependency extraction and parameter reasoning (e.g., GPT-4 and GPT-3.5).
  • Operation-cognition knowledge base, systematically categorizing five major types of cognitive dependencies and their subcategories through inference rules.
  • Rule-driven UI extraction and iterative task recognition to support modular task analysis.

Research Outcomes

• Specific Outcomes:

  1. TaskMind creatively utilizes task graphs and cognitive dependencies to significantly improve the generalization capabilities of automation for tasks with multiple parameter dependencies.
  2. The operation-cognition knowledge base systematically organizes diverse cognitive dependency types across 39 complex tasks, demonstrating its academic value and practical potential.
  3. A reflection module was designed and implemented, enabling the task model to evolve continuously through user feedback, making the system more adaptable to personalized needs.

• Comparative Advantages Over Existing Solutions:

  1. Enhanced Generalization Capability: By explicitly modeling cognitive dependencies, TaskMind dynamically adjusts parameters when task objectives change, whereas existing methods often rely solely on "record-replay" mechanisms.
  2. Flexibility and Transparency: The task graph design visualizes complex task logic, making it easier for users to understand the automation process and make targeted modifications.
  3. Improved Automation Success Rate: Compared to traditional end-to-end LLM-based methods, TaskMind increases task success rates from 79.5% (unmodified) to 92.3% (after user modifications), significantly reducing failure rates.

• Experimental or Evaluation Results:

  1. Simulation experiments show that TaskMind achieves high accuracy in identifying semantic dependencies, with an F1 score of 82.6%.
  2. User studies reveal that in 120 experiments, task completion time was reduced by 66%, and subjective satisfaction, reliability, and sense of control were significantly higher compared to alternative methods.
  3. In personalized task tests, TaskMind performed well across various real-world tasks (e.g., data processing, learning support, and document management), achieving a task success rate of 75%.

• Limitations and Future Directions:

  1. Application Scope Limitations: TaskMind currently supports only a limited set of GUI applications (e.g., Excel, Chrome, and file managers) and struggles with dynamic web structures and certain desktop applications.
  2. Fixed Path Issue: The automation process follows a fixed operation path, supporting only simple task flows and not suitable for complex workflows or multi-branch tasks.
  3. Automated Task Segmentation: TaskMind still requires users to manually mark task boundaries; future work could integrate rules or machine learning algorithms to achieve automated segmentation.
  4. User Interface Enhancements: Users have expressed a desire for a lower learning curve and more flexible switching between natural language and task graph interfaces.

Through the research and development of TaskMind, this paper provides a new perspective on GUI task automation, combining cognitive theory and artificial intelligence technologies to demonstrate how automation tools can drive innovation in human-computer collaboration by deeply modeling user intentions. This work offers important guidance for the future development of complex task automation and explainable design.