ACKnowledge: A Computational Framework for Human Compatible Affordance-based Interaction Planning in Real-world Contexts

Research Background and Problems

• Problems and Challenges:

  1. Intelligent agents (e.g., household robots) operating in environments shared with humans often need to plan interactions based on object affordances. However, ensuring these interactions are both physically feasible and aligned with human social values and psychological expectations remains an unresolved issue.
  2. Most existing methods focus on static attributes or reasoning based on vision and language, often neglecting the combined effects of complex physical, intrapersonal, and interpersonal contextual factors in dynamic environments.
  3. Beyond delivering acceptable task outcomes, there is a lack of planning systems that are process-transparent and capable of meeting users' personalized needs, making efficient user understanding and interaction difficult to achieve.

• Significance: Interactions that do not align with human practices and social expectations may damage objects or the environment and significantly reduce user satisfaction and trust in intelligent agents.

• Research Motivation and Related Work:

  1. This paper addresses the need to solve these issues to advance affordance-based human-agent interaction planning.
  2. Drawing from cognitive science and HCI (e.g., 4E cognition theory and dual-process theory), it emphasizes that the cognitive architecture of intelligent agents should align with human decision-making processes.
  3. Previous research has limitations, including insufficient contextual modeling, lack of dynamic adaptability, and inadequate support for user explainability.

Solution

• Proposed Solution:

  • This paper introduces "ACKnowledge," a computational framework for dynamic environments that integrates dynamic knowledge graphs, vision-language models (VLM), and large language models (LLM) to support affordance-based human-compatible interaction planning.
  • The framework mimics the human "dual-process cognitive architecture":
    • Intuitive Thinking (System 1): Generates theoretically feasible interaction plans through dynamic knowledge graphs.
    • Analytical Thinking (System 2): Validates and ranks plans based on physical, intrapersonal, and interpersonal contexts.

• Innovations:

  1. Incorporates physical, intrapersonal, and interpersonal contextual factors into dynamic knowledge graph modeling.
  2. Utilizes large language models to support multi-stage reasoning and plan generation, enhancing system explainability and user interaction capabilities.
  3. Provides a transparent, negotiable, and personalized planning process.

• Core Technologies and Implementation Steps:

  1. Knowledge Modeling: Constructs an "affordance learning knowledge graph" based on the ATOMIC2020 dataset, modeling high-frequency relational weights for objects and their functions.
  2. Context Awareness: Employs vision-language models to capture dynamic environmental factors (e.g., object quantity and occupancy status) and update graph weights.
  3. Inference-Based Plan Generation: Uses Retrieval-Augmented Generation (RAG) techniques for task-critical requirement analysis, candidate plan generation, and ranking.
  4. User Interaction:
     • Graphical User Interface (GUI) allows users to adjust personalized configurations, such as object usage permissions.
     • Conversational User Interface (CUI) enables users to correct planning errors through language input.

Research Outcomes

• Specific Outcomes:

  1. ACKnowledge outperformed two baseline models in execution effectiveness and user satisfaction, generating plans that are more socially adaptive and respectful of usage permissions.
  2. The integration of intuitive and analytical reasoning significantly improved the explainability and acceptability of the plans.
  3. The UI interfaces were deemed efficient and usable by users, particularly in tasks involving personalization and error correction.

• Advantages Over Existing Solutions:

  1. Places greater emphasis on the psychological and social dimensions of human perception (e.g., ownership and social etiquette) compared to traditional systems.
  2. Provides multi-stage reasoning and context adaptability in complex dynamic scenarios.
  3. Enhances transparency and user interaction efficiency, making the system more aligned with human operational habits.

• Experimental and Evaluation Results:

  1. Simulation Experiments: ACKnowledge achieved a 98.78% success rate in plan executability, significantly outperforming baseline models.
  2. User Subjective Feedback: In evaluations of metrics such as success rate, physical validity, and social adaptability, ACKnowledge received higher user ratings.
  3. Personalization and Error Correction Testing:
     • Case studies demonstrated that users could understand and adapt to ACKnowledge's planning process.
     • User-modified plans not only improved actual tasks but were successfully applied to similar tasks.

• Limitations and Future Work:

  1. Limitations:
     • The knowledge graph construction is based on a single dataset, ATOMIC2020, which may result in incompleteness.
     • Case studies used AR simulations instead of real robots, excluding issues related to physical execution.
     • The sample size was small, and the user group was relatively homogeneous.
  2. Future Directions:
     • Integrate more data sources to enhance the richness of the knowledge graph.
     • Deploy on real robots to achieve higher spatiotemporal interaction planning capabilities.
     • Expand to multi-user scenarios to study how to coordinate diverse user needs in shared environments.
     • Design more multimodal sensing capabilities and diverse sensor modules to improve environmental understanding accuracy.

Through the research on ACKnowledge, this framework provides foundational ideas and technical support for the application of next-generation intelligent agents in dynamic human environments, showcasing the potential to promote human-agent symbiosis.