Knowledge Graph Completion-based Question Selection for Acquiring Domain Knowledge through Dialogues

Title of the Paper

Knowledge Graph Completion-based Question Selection for Acquiring Domain Knowledge through Dialogues

Paper Information

• Subject Area: Natural Language Dialogue Systems, Knowledge Graphs, Knowledge Acquisition
• Keywords: Dialogue Systems, Knowledge Acquisition, Knowledge Graph Completion, Question Selection, User Interaction, Subjective Evaluation, Fuzzy Knowledge Modeling

Research Background and Problem

• Identified Problems/Challenges:

  • Constructing a perfect knowledge base for a specific domain is highly challenging.
  • Knowledge graphs, as the knowledge base for dialogue systems, often suffer from incompleteness.
  • When calibrating knowledge acquisition through natural language dialogues with users, there is a risk of selecting questions containing incorrect knowledge, which reduces users' willingness to engage in dialogue.

• Significance of the Research:

  • Natural language dialogues provide an opportunity to enhance the system's knowledge base through user-shared knowledge while enriching the user interaction experience.
  • Completing knowledge graphs is crucial for the proper functioning of information service systems, such as recommendation or question-answering systems.

• Related Work:

  • Existing studies have explored how dialogue systems utilize knowledge graphs to understand user intent and generate recommendations or responses, but these methods are mostly based on static knowledge graphs.
  • Some research has attempted to acquire lexical or ontological knowledge through dialogues, but there has been little discussion on how to avoid negative user impressions and ensure the correctness of question generation.

Proposed Solution

• Method/Framework:

  • A framework based on Knowledge Graph Completion (KGC) is proposed to predict potentially correct links and use them to select questions for the system to ask users.
  • Questions are generated based on links with high completion scores, while links with low scores are avoided to reduce potential negative feedback from users.

• Innovations:

  • Optimizing the question selection strategy based on KGC output scores.
  • Improving the reliability of KGC output scores through two training phase modifications:
    1. Connecting unlinked entities using substrings of entity names.
    2. Restricting the range of negative sampling to focus the training process on valid samples.

• Implementation Steps:

  1. Predict unknown links in the knowledge graph, generating candidate triples and their associated scores.
  2. Select high-scoring triples to generate natural language questions for user verification.
  3. Improve model training to enhance score reliability, including substring expansion and constrained negative sampling.
  4. Add user-confirmed links to the knowledge graph through interaction for continuous improvement.

• Key Technologies Used:

  • Knowledge Graph Completion method ComplEx for generating embeddings and predicting links.
  • Designed enhancements such as "substring generation" and "negative sampling restriction."

Research Outcomes

• Specific Results:

  • Experimental results show that the improved ComplEx model (Sub+NF) significantly outperforms the baseline model in the Hits@1 metric.
  • In the food and restaurant domain, crowdsourced user surveys revealed that the system-generated questions were more reasonable and reduced potential negative user impressions.

• Comparison with Existing Solutions:

  • Compared to unmodified KGC methods, the added Sub step significantly improved link prediction accuracy in incomplete knowledge graphs.
  • In user testing, the improved model generated more "correct" and "obviously correct" questions (44.0% positive ratings, significantly higher than the baseline's 28.2%).

• Experiments and Evaluation Results:

  • Evaluation Subjects: Prediction accuracy (Hits@1) was calculated through cross-validation.
  • Experimental Data: A manually constructed food and restaurant domain knowledge graph (containing 7,304 entities, 14 relations, and 27,914 triples).
  • Evaluation Metrics:
    • The improved model showed Hits@1 improvements across multiple relation types, with overall accuracy increasing from 24.2% to 36.1%.
    • Subjective user evaluations showed a significant improvement in the correctness of template questions, with positive ratings for Sub+NF questions 15.8% higher than the baseline.

• Limitations and Future Directions:

  1. Limitations:

     • The improvement method heavily relies on the similarity of incomplete entity names, making it suitable for specific domains.
     • "Obviously correct" questions may be perceived as repetitive and tedious by users, requiring further optimization of question strategies.

  2. Future Directions:

     • Integrate the framework into dialogue systems and test its impact on actual user experience.
     • Explore dynamic adjustments to dialogue strategies to make questions better aligned with user preferences.
     • Investigate the impact of new knowledge graph characteristics (e.g., different language environments) on the applicability of the solution.