Increasing the Speed and Accuracy of Data Labeling Through an AI Assisted Interface

Title of the Paper

Increasing the Speed and Accuracy of Data Labeling Through an AI Assisted Interface

Paper Information

• Subject Area: Artificial Intelligence and Human-Computer Interaction; AI Collaboration in Data Labeling
• Keywords: AI Assistance, Data Labeling, Human-Computer Interaction, Machine Learning, Intelligent Interface Design

Research Background and Problem

• Identified Problems or Challenges:

  1. Data labeling is a critical yet tedious step in supervised learning, requiring extensive human decision-making.
  2. Labeling tasks with large-scale label sets (ranging from tens to thousands of labels) are highly complex, often resulting in reduced accuracy and increased time consumption.
  3. Existing studies primarily focus on the effects of AI assistance in binary classification scenarios, with limited exploration of AI assistance in multi-label scenarios.

• Significance of the Problem:

  • As AI systems are increasingly utilized, the quality of labeled datasets directly impacts the performance of machine learning models. Improving the accuracy and efficiency of data labeling has become a critical need.

• Research Motivation and Related Work:

  • Human-AI collaboration and AI-assisted technologies have been proven to enhance the accuracy of binary decision-making tasks.
  • Existing literature has discussed the impact of transparency, explainability, and trust calibration on AI-assisted decision-making, but there remains a significant gap in the design of AI assistance and user behavior studies for multi-label tasks.

Proposed Solution

• Method or Solution:
  The authors propose an AI-assisted data labeling system based on a semi-supervised learning algorithm to predict the potential label distribution of samples and provide labeling assistance through the following methods:

  1. Offering label recommendations;
  2. Reducing the decision space for labelers, focusing only on the most likely labels.

• Innovative Contributions:

  1. Addressing complex decision-making problems in multi-label scenarios;
  2. Proposing a labeling interface design that ranks and reduces the decision space based on predicted probabilities;
  3. Designing comprehensive experiments to evaluate the specific impact of AI assistance on labeling accuracy and speed.

• Implementation Steps and Key Techniques:

  1. Using a semi-supervised learning algorithm based on Label Spreading to predict the label distribution of unlabeled samples.
  2. Displaying the top 5 most likely labels on the labeling interface, ranked by predicted probabilities.
  3. Providing a "view all labels" feature to allow user intervention.
  4. Designing experiments to evaluate the assistance effect by comparing the performance of baseline, weak AI, and strong AI groups.

Research Findings

• Specific Outcomes:

  1. AI assistance significantly improved labelers' accuracy: accuracy increased by 6% compared to the baseline group.
  2. AI assistance significantly reduced the time required to complete labeling tasks, especially when the needed label was among the AI-recommended "top 5 labels."
  3. There was no significant difference in accuracy between weak AI and strong AI assistance: users were able to identify incorrect predictions and make independent decisions.
  4. The distribution of AI recommendation confidence (predicted probabilities) influenced user behavior: high-confidence incorrect predictions from weak AI reduced user trust but increased critical judgment.

• Advantages Compared to Existing Solutions:

  1. Applicable to more complex multi-label labeling scenarios, achieving higher efficiency than traditional methods.
  2. Effectively combines label recommendation and decision space reduction techniques, enhancing user experience.

• Experimental and Evaluation Results:
  The average accuracy of users in the weak AI-assisted group was 0.79, in the strong AI-assisted group was 0.78, and in the baseline group was only 0.72. The labeling time in the assisted groups was significantly shorter than in the baseline group.

• Limitations and Future Directions:

  1. The current study only covers one labeling task and should be extended to different task types and difficulty scenarios.
  2. The experiments involved non-expert labelers; further exploration is needed for the effects of assistance in expert labeling contexts.
  3. The presentation of labels in the user interface (e.g., whether to fix spatial order) requires further optimization and research.
  4. The potential impact of different AI system confidence distributions on user behavior warrants deeper exploration.