ONYX: Assisting Users in Teaching Natural Language Interfaces Through Multi-Modal Interactive Task Learning

Title of the Paper

ONYX: Assisting Users in Teaching Natural Language Interfaces Through Multi-Modal Interactive Task Learning

Paper Information

• Subject Area: Human-Computer Interaction, Natural Language Interfaces, Task Learning
• Keywords: Interactive Task Learning, End-User Development, Natural Language Interfaces, Data Visualization Tools, User Interface Programming

Research Background and Problem

• The widespread adoption of Natural Language Interfaces (NLIs) enables users to interact with computing systems more naturally. However, their one-size-fits-all design often results in user attrition, especially when the system fails to understand new Natural Language (NL) commands provided by users.
• Current NLIs can be extended through users' procedural and declarative knowledge, but the teaching process faces several challenges, such as:
  1. Understanding the complexity and diversity of user demonstrations and generalizing them to specific goals.
  2. Users lacking insight into how NLIs interpret their inputs.
  3. Frequent trial-and-error during the teaching process.

Research Motivation and Related Work

• Existing Interactive Task Learning (ITL) methods primarily focus on generalizing declarative knowledge, while the ability to generalize procedural knowledge remains insufficient.
• Data visualization tools serve as an ideal research scenario due to their inherent ambiguity and the need for multi-step operations to achieve goals.
• This study aims to explore how multi-modal ITL (combining natural language programming and demonstration programming) can improve NLI learning capabilities.

Solution

Core Methodology

• Propose ONYX, an intelligent agent system that utilizes a multi-modal approach (natural language programming and task demonstration) to learn NL inputs and extract procedural and declarative knowledge from user operations.
• Key design features:
  1. Suggestions: Provide operation recommendations based on known concepts and user-defined procedures.
  2. Follow-up Questions: Pose clarifying questions during user demonstrations to eliminate ambiguity.
  3. Visual and Textual Aids: Display system understanding through visual and textual guidance during critical teaching stages, enabling users to provide intuitive feedback and corrections.

Implementation Steps

1. User-Centered Design:
   • Employ participatory design (10 participants) to iteratively develop six versions for continuous refinement.
2. Core Interaction Workflow:
   • The system identifies known and unknown parts of new NL inputs and provides teaching suggestions.
   • Users complete the unlearned parts through direct operations, while the system enhances ambiguity resolution via follow-up questions.
   • All processes are represented through visual scripts, allowing users to intuitively adjust the system's understanding.
3. System Architecture:
   • Web-based client-server architecture, including a natural language parser and ITL agent module.

Research Outcomes

Experiments and Evaluation

• Experimental Design:
  • Conduct online experiments with 42 participants, divided into an experimental group with key features and a control group with basic features.
  • Participants solve three typical NL input tasks that the system initially fails to understand, emphasizing the use of suggestions and follow-up questions.
• Performance Improvement:
  • The experimental group with suggestions and follow-up questions achieved significantly higher teaching accuracy (median 93.3%, control group 73.3%, p < 0.001).
  • No significant difference in time cost between the two groups.
• Behavioral Analysis:
  • Suggestions helped users quickly identify unknown parts of the system's learning and establish operational workflows.
  • Follow-up questions effectively reduced ambiguity-related errors (by 65.4%), though users required some learning time to familiarize themselves with the assistance features.

Key Advantages

• ONYX enhances the efficiency and accuracy of teaching new NL inputs through multi-modal ITL.
• The introduced features directly improve users' understanding of system knowledge and the teaching process, reducing trial-and-error costs.

Limitations and Future Directions

1. Ambiguity Issues:
   • The current system cannot handle the ambiguity of the same NL input in different contexts. Future work should model context using situational information (e.g., user behavior history, operation timing).
2. Semantic Understanding:
   • The system's understanding of keywords, synonyms, and antonyms is limited. Integrating semantic knowledge bases like WordNet could enhance semantic parsing.
3. Cold-Start Problem:
   • The system's initial knowledge is limited. Plans include integrating existing NLI tools' language models to mitigate this issue and combining them with large-scale NL processing models based on machine learning.

Conclusion

ONYX proposes an effective collaborative approach for task learning and personalized NL input scenarios. Through intelligent multi-modal interaction, it significantly improves users' experience and effectiveness in teaching NL systems, offering valuable insights for the design of future personalized NLI systems.