PColorizor: Re-coloring Ancient Chinese Paintings with Ideorealm-congruent Poems

Title of the Paper

PColorizor: Re-coloring Ancient Chinese Paintings with Ideorealm-congruent Poems

Paper Information

• Domain: Digital Color Restoration of Ancient Chinese Paintings
• Keywords: Ancient Chinese Art, Color Restoration, Poem-Painting Integration, Visualization Analysis, Cultural Heritage Preservation, Deep Learning, User Interaction

Research Background and Problem

• Problem or Challenge:

  • Ancient Chinese paintings have suffered significant color fading due to improper preservation. Manual color restoration is time-consuming and labor-intensive, yet it is a crucial part of cultural preservation and inheritance.
  • Current automated methods, such as image style transfer or colorization techniques, have been applied to restoration. However, the "black-box" nature of deep learning models makes it difficult to assess the credibility of the restored colors.
  • Additional challenges include the difficulty of collecting reference paintings and the complexity of tracing color evolution.

• Research Significance:

  • Promoting digital virtual color restoration helps extend the cultural life of ancient paintings.
  • Preserving the cultural emotion or "ideorealm" expressed in paintings helps viewers relive the historical significance conveyed by the artwork.

• Research Motivation and Related Work:

  • The traditional relationship of "ideorealm congruence" between Chinese ancient paintings and poetry provides potential clues for color references.
  • While many deep learning-based image processing methods exist, how to effectively integrate cross-modal information from poetry and paintings to assist color restoration remains underexplored.

Solution

• Method or Solution:

  • An interactive system named PColorizor is proposed, combining deep learning models (e.g., CLIP) with novel visualization techniques to support efficient and intuitive color restoration.
  • By leveraging the "poem-painting congruence" principle, ideorealm-congruent poems are used to identify reference paintings related to the target artwork.
  • The system design integrates a query construction module, a poem-painting congruence matching module, a color analysis module, an ideorealm analysis module, and a user-guided colorization module.

• Innovations:

  • Introduced a CLIP-based cross-modal model to extract implicit color clues from poems to assist in matching reference paintings.
  • Proposed a novel visualization scheme based on the "mountain shape" metaphor to display the temporal color evolution and distribution of reference paintings.
  • Incorporated both automatic and semi-automatic colorization operations in the system, supporting user interaction to reduce restoration time.

• Implementation Steps and Techniques:

  1. Cross-modal Learning: Used the CLIP model to establish cross-modal matching between poems and paintings, fine-tuned with a dataset of 11,000 pairs of ancient paintings and poems.
  2. Color Analysis and Visualization:
     • Created a "timeline view" combining mountain shape metaphors, bar charts, and scatter plots to explore the color evolution of paintings.
     • Introduced heatmaps to visualize the deep model's understanding of ideorealm.
  3. User Interaction Tools: Provided tools for fuzzy region selection, text-based combination searches, and color transfer from reference paintings.
  4. Experiments and Validation: Conducted two case studies to validate the system's practicality and collected feedback in collaboration with domain experts.

Research Outcomes

• Specific Outcomes:

  • Developed a complete interactive system integrating a poem-painting congruence module, a visualization interface, and a user-guided colorization method, successfully aiding in painting color restoration.
  • Created an innovative timeline visualization technique to extract reliable color schemes from a large number of reference paintings.
  • Demonstrated through case studies that the system significantly improved the efficiency of color restoration and the accuracy of reference painting searches.

• Comparative Advantages:

  • Compared to existing methods, PColorizor significantly reduces manual operation requirements and provides additional color clues derived from poetic ideorealm, which are difficult to uncover using traditional methods.
  • Offers a dynamic and exploratory way to connect reference paintings, addressing the limitations of existing systems in handling large-scale color references.

• Experimental or Evaluation Results:

  • Case studies showed that experts could more efficiently extract useful color information from reference paintings using the system.
  • The system accurately interpreted different types of ideorealm expressed in poetry, providing reliable model assistance for color restoration.

• Limitations and Future Directions:

  1. Data Scale Limitations: The current dataset size limits the ability to support more complex ideorealm expressions.
  2. Model Limitations: Although the cross-modal learning model performs well, there is room for improving accuracy in rare poem-painting ideorealm combinations.
  3. Visualization Scalability: The current visualization scheme may face spatial and interaction complexity issues when dealing with more reference paintings.
  4. Future Improvements: Introduce LLMs (e.g., ChatGPT) to enhance the understanding of user-customized descriptions; optimize user experience by presenting richer historical contexts with finer-grained dynastic segmentation.