VAL: Interactive Task Learning with GPT Dialog Parsing

Title of the Paper

VAL: Interactive Task Learning with GPT Dialog Parsing

Paper Information

• Field of Study: Human-Computer Interaction, Task Learning, Natural Language Processing
• Keywords: Hierarchical Task Networks, GPT, Large Language Models (LLMs), Neuro-Symbolic AI, Hybrid AI

Research Background and Problem

• Background:
  • Interactive Task Learning (ITL) is a method for incrementally acquiring task knowledge through natural human interaction, aiming to enable computers to learn general-purpose tasks from human instruction via language.
  • Existing ITL systems rely on syntactic-semantic parsing, which often results in unnatural interactions and fragility due to parsing errors. Large Language Models (LLMs) are more flexible in semantic understanding but lack incremental learning capabilities and interpretability.
• Problem or Challenge:
  • How to combine the advantages of LLMs with the transparency and scalability of classical symbolic learning to achieve ITL.
  • The fragility of language parsing in current ITL systems makes it difficult to support complex and natural conversational scenarios.
  • Insufficient capabilities in coreference resolution, semantic ambiguity handling, and task generalization from language.
• Significance of Research:
  • Advancing ITL systems to enable natural language interaction with humans while maintaining interpretability and generalizability.
  • Improving human-computer interaction experiences, making generalization and human guidance feasible.
• Research Motivation: Proposing an effective neuro-symbolic hybrid framework to address the aforementioned challenges.

Solution

• Proposed Method:

  • Develop an ITL system named VAL (Verbal Apprentice Learner) that combines GPT-family LLMs with a symbolic task learning framework.
  • The system focuses on learning hierarchical task knowledge through natural language dialogue, leveraging LLMs for specific subtasks (e.g., predicate selection) rather than relying on LLMs end-to-end.

• Innovations:

  1. Unique neuro-symbolic hybrid architecture:
     • Utilize GPT for semantic parsing and subtask processing in dialogues (e.g., predicate extraction and parameter selection).
     • Represent task structures using classical Hierarchical Task Networks (HTN).
  2. Reduce cascading errors and improve interaction usability through confirmation dialogues and undo operations.
  3. Enable incremental learning and task generalization, supporting teaching through natural language dialogue.
  4. Provide user-friendly visualization of task knowledge.

• Implementation Steps:

  1. Use the "VALgorithm" algorithm to structure natural language input:
     • Decompose user input into atomic steps;
     • Employ GPT for predicate matching, parameter mapping, and validation.
  2. Provide a Hierarchical Task Networks (HTN) framework to learn task hierarchies.
  3. Implement error correction through a user interaction confirmation interface.
  4. Design a real-time interactive interface for user teaching and task monitoring.
  5. Test the system's learning performance in video game environments (e.g., Overcooked-AI).

Research Outcomes

• Specific Results:
  • The VAL system can incrementally learn complex hierarchical task structures through minimal natural language interaction (e.g., tasks in the Overcooked game).
  • Offers intuitive knowledge representation (e.g., real-time visualization and confirmation dialogue interface), enhancing user experience.
• Advantages:
  • Compared to existing ITL systems, VAL significantly improves robustness in handling grammatical errors, unnatural expressions, and ambiguous language.
  • Combines LLMs' strengths in flexible language understanding with the interpretability of HTN-based symbolic learning.
  • Supports richer and more natural user interaction, optimizing user engagement.
• Experimental Results:
  • In a user study involving 12 participants, VAL demonstrated the ability to complete most task learning scenarios.
  • Users generally described the system as "easy to use" and "natural," though frequent confirmation dialogues were noted as potentially tedious.
  • High average accuracy was observed across different submodules, particularly in input segmentation and parameter mapping.
• Limitations and Future Directions:
  • Frequent confirmation dialogues increase user fatigue; future work could aim to reduce unnecessary confirmations.
  • Enhance environmental awareness and automatic task generation to reduce user teaching burden.
  • Expand VAL's capabilities to other task modules and multimodal interactions (e.g., visual and gesture-based).
  • Replace commercial LLMs with open-source models to improve system transparency and scalability.