Spatial Labeling: Leveraging Spatial Layout for Improving Label Quality in Non-Expert Image Annotation

Document Title

Spatial Labeling: Leveraging Spatial Layout for Improving Label Quality in Non-Expert Image Annotation

Document Information

• Field of Study: Human-Computer Interaction and Image Annotation Optimization
• Keywords: Manual Image Annotation, Non-Expert Annotators, Spatial Layout, Interface Design, Label Quality Improvement, Human-Computer Interaction, User Study

Research Background and Problem

• Problem or Challenge: In image annotation tasks, due to the limited availability and high cost of expert annotators, non-expert annotators (e.g., crowdworkers) are often employed. However, non-expert annotators lack domain knowledge, which can lead to low annotation quality and high error rates.
• Importance: Image annotation is a critical step in building machine learning applications for images, and annotation quality directly impacts model performance. Therefore, finding ways to help non-expert annotators improve annotation quality is an urgent issue.
• Research Motivation and Related Work: Traditional annotation tools fail to effectively support non-expert annotators in observing and organizing similarities and differences between images, resulting in high annotation error rates. Related work has mainly focused on expert annotation or improving annotation quality through crowd collaboration, with little attention given to optimizing tasks performed independently by non-expert annotators.

Solution

• Method or Solution: A spatial layout-based annotation interface design, called "Spatial Labeling," is proposed. This method allows annotators to arrange images and labels in an open space to express conceptual similarities between images, forming a temporary organization for annotation before final labeling.
• Innovations:
  1. Introducing the concept of spatial layout into non-expert image annotation tool design.
  2. Providing a workspace to help annotators observe and organize image similarities and differences.
  3. Increasing annotators' confidence in label selection.
• Key Technologies and Implementation Steps:
  1. Annotators drag and arrange images and related labels in an open space to form conceptual clusters.
  2. Final labels are assigned to images based on the results of the layout organization.
  3. The system includes a confidence state indicator, requiring annotators to express their confidence level in label selection after completing the annotation.
  4. Annotators can modify assigned labels or adjust the layout at any time.

Research Outcomes

• Specific Results:
  • The spatial layout interface significantly reduced annotation error rates (from 43.50% in traditional layouts to 37.63%).
  • Annotators' confidence in label selection increased (from 47.13% in traditional layouts to 59.63%).
• Advantages Over Existing Solutions:
  • Improved the quality of tasks handled independently by non-expert annotators without relying on experts or other collaborators.
  • Particularly effective in complex and ambiguous annotation scenarios.
• Experimental or Evaluation Results:
  • In the user study, the annotation task involved 50 dog images and 20 dog breed labels. Compared to traditional non-spatial layout interfaces, the spatial layout interface provided higher annotation accuracy and confidence.
  • Spatial layout was significantly effective for annotation tasks with moderate ambiguity in dog breeds but had limited effectiveness for completely clear or highly ambiguous image categories.
  • Although the spatial layout interface scored lower in user satisfaction, its functional performance was outstanding.
• Limitations and Future Directions:
  • The reasons behind the improvement in actual label quality are not fully clear, possibly involving learning effects or more cautious label selection.
  • The experiment scale was small, including only 20 labels and 50 images; future studies need to expand to large-scale annotation tasks for validation.
  • Currently focused on image classification tasks, future work will extend to image object detection tasks to further explore the potential of spatial layout in complex image annotation.
  • Plans to develop collaborative annotation interfaces and incorporate external reference resources (e.g., web search) to optimize non-expert annotation quality.

Conclusion

The authors proposed a spatial layout-based annotation method that improves image annotation quality through the "organizational observation process" of non-expert annotators. Experimental results show that the spatial layout interface not only performs well in annotation quality but also enhances annotators' confidence. Despite users preferring traditional layout interfaces, the spatial layout interface demonstrates promising applications under complex annotation conditions. This study provides valuable insights for future annotation tool design and optimization of non-expert annotation quality.