No Pixel Left Behind: Filling Gaps in Anime Colorization

Paper Title

No Pixel Left Behind: Filling Gaps in Anime Colorization

Publication Info

• Topic area: AI-powered tools for digital painting in anime production workflows.
• Keywords: anime colorization, gap detection, deep learning, creativity support tools, user study, AI-powered assistance, professional workflows, human-AI collaboration, production pipelines, visual quality.

Background and Problem

• Problem / challenge: Small unpainted regions (“gaps”) in anime colorization workflows are difficult to detect and fill, requiring repetitive manual effort. Existing tools fail to address this challenge adequately.
• Significance: These gaps impact visual quality and production efficiency, necessitating costly retakes in professional workflows. Addressing this issue can reduce labor costs and improve adherence to strict quality standards.
• Motivation and related work: Prior research has explored automatic colorization methods and AI-powered creativity tools but has largely overlooked the specific challenge of gap-filling in anime production. This paper builds on these insights to address the gap-filling problem using a domain-specific approach.

Solution

• Proposed approach: GapFill, a specialized tool for detecting and filling small unpainted regions in anime colorization workflows, integrating deep learning-based color prediction and user-friendly interaction techniques.
• Novelty:
  1. Automatic detection and highlighting of unpainted gaps using circular markers.
  2. Deep learning-based color prediction leveraging local context and flat-color characteristics of anime images.
  3. Interactive features such as pop-up magnification, in-circle color correction, and batch application of suggested colors.
  4. Seamless integration into professional workflows with human-in-the-loop design principles.
• Procedure and key techniques:
  • Detect gaps using BFS-based segmentation and highlight them with adjustable thresholds.
  • Predict colors using a U-Net model that outputs likelihood maps for neighboring regions.
  • Enable manual correction via drag-and-drop color selection and batch application of AI-suggested colors.
  • Evaluate usability and performance through structured tasks and user studies with professional colorists.

Results

• Concrete findings:
  • GapFill improved task completion times in gap-filling tasks (Task B1: 45.15s vs. 57.69s; Task B2: 51.91s vs. 66.27s) and eliminated overlooked gaps.
  • Prediction accuracy of 81.68% on unseen datasets, outperforming a naive baseline (37.02%).
  • Average inference time per patch: 74ms on NVIDIA RTX 6000 Ada Generation.
• Advantage over baselines:
  • Faster task completion and higher effectiveness in gap detection and filling compared to conventional tools like Black Light Method and Leftover Pen.
  • Enhanced usability through intuitive interactions and reduced manual effort.
• Experiments / evaluation:
  • User study with 13 professional colorists comparing GapFill against baseline tools across three tasks (coloring from scratch, gap detection and filling, and automated color prediction evaluation).
  • Metrics: task completion time, number of overlooked gaps, subjective usability ratings, and qualitative feedback.
• Limitations and future work:
  • Accuracy issues in detailed regions like pupils and clustered gaps.
  • Need for integration with existing tools like Black Light Method.
  • Future directions include improving prediction accuracy, handling anti-aliased line art, and redefining accuracy metrics based on perceptual quality.

Summary

This paper introduces GapFill, a tool designed to address the challenge of small unpainted gaps in anime colorization workflows. By combining automatic gap detection, deep learning-based color prediction, and user-friendly interaction techniques, GapFill significantly improves efficiency and usability in gap-filling tasks. A user study with professional colorists demonstrated its advantages over conventional tools, highlighting its potential for integration into existing workflows. While prediction accuracy remains a conditional factor for usability, the tool’s features and human-in-the-loop design emphasize user control and adaptability, making it a valuable addition to professional pipelines.