Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels

Title of the Paper

Human-LLM Collaborative Annotation Through Effective Verification of LLM Labels

Paper Information

• Domain: Data Annotation in Human-AI Collaboration for Natural Language Processing
• Keywords: Human-AI Collaborative Annotation, LLM Annotation, Self-Explanation, Text Annotation, NLP

Research Background and Problem

• Problems and Challenges:

  • Large Language Models (LLMs) perform well in many natural language processing tasks but are prone to errors and biases when generating annotations for complex or domain-specific tasks.
  • Fully replacing human annotation with LLMs raises concerns about accuracy and reliability.

• Significance:

  • Data annotation is a critical component of machine learning development, but it is costly and time-consuming.
  • Building reliable and efficient annotation mechanisms is crucial, especially for improving annotation quality and addressing model bias.

• Research Motivation and Related Work:

  • LLMs offer a flexible and cost-effective annotation solution and can assist in quality verification through self-explanation.
  • Model explanations can enhance human understanding of LLM-generated labels.
  • Current research has explored multiple directions for verifying the quality of LLM labels and supporting annotation for complex tasks.

Proposed Solution

• Proposed Method or Solution:

  • This paper introduces a multi-step framework (Lapras) for collaborative data annotation by humans and LLMs, consisting of the following steps:
    1. LLM generates labels and explains its decisions.
    2. A verifier model scores the quality of the LLM-generated labels.
    3. Humans re-annotate a subset of low-scoring labels.

• Innovations:

  • Introducing an additional verification step to score LLM-generated labels, reducing the waste of human effort on reviewing correct labels.
  • Leveraging LLM-generated explanations to enhance verifier effectiveness and help humans better understand the generated labels.

• Implementation Steps and Key Techniques:

  1. LLM Annotation: Using zero-shot or few-shot learning methods, labels and explanations are generated through instructions and example templates.
  2. Verifier Scoring: An independent binary classification model (e.g., Random Forest, SVM, or language models like BERT) scores the quality of LLM labels by combining input sample features, label features, and explanation features.
  3. Human Review: Labels identified as potentially erroneous by the verifier are assigned to humans for re-annotation. Experiments investigate whether and how LLM-generated assistant information should be presented during human annotation.

Research Outcomes

• Specific Results:

  • The verifier performed well across six text classification datasets, accurately identifying erroneous labels.
  • Experiments demonstrated that a verifier could be effectively trained with only 200 samples.

• Comparative Advantages:

  • Compared to a fully manual annotation system, this framework significantly reduces annotation costs while avoiding errors that may arise from fully relying on LLM-generated labels.

• Experimental and Evaluation Results:

  • The verifier's accuracy in identifying erroneous labels significantly outperformed baseline methods (e.g., scoring based on LLM prediction confidence).
  • Human-AI collaboration experiments showed that presenting LLM labels and explanations to humans could improve annotation quality in certain cases, though poor-quality LLM explanations might mislead human annotators.

• Limitations and Future Directions:

  • The verifier's performance depends on gold-standard labeled datasets, and its adaptability to cross-dataset tasks needs improvement.
  • Improving the quality of LLM-generated explanations is a key area for future research, especially for domain-specific or complex tasks.
  • Further exploration is needed to develop simplified and optimized interaction interfaces for presenting explanations.

Overall, the Lapras framework establishes an efficient method for human-AI collaborative annotation. Through performance validation and user experiments, it demonstrates potential applications in data annotation while offering design directions for improving model verification and human collaboration.