Reviving Mural Art through Generative AI: A Comparative Study of AI-Generated and Hand-Crafted Recreations

Research Background and Issues

• Issues or Challenges: Traditional methods for manually creating virtual reality (VR) mural scenes are both time-consuming and labor-intensive, requiring specialized knowledge in art and digital modeling. These methods struggle to scale when dealing with complex and large-scale scenes, thereby limiting the scope of digital preservation and dissemination of cultural heritage.
• Significance: Murals are an important form of cultural heritage that encapsulate the history, religion, and artistic expressions of ancient civilizations. However, access to these artifacts is often restricted by geographical limitations, maintenance cycles, or their complex locations, posing risks to the preservation of historical heritage. VR offers new ways to interact with and protect these artifacts, enhancing public engagement and understanding.
• Research Motivation and Related Work: While some studies have explored manually converting 2D murals into 3D scenes, the high level of expertise required and the need for manual intervention make such approaches costly and time-intensive. Recent advancements in Generative AI (GenAI) provide automated and scalable solutions for creating 3D scenes from 2D murals. However, current applications often fail to meet fidelity requirements, particularly in preserving original colors, textures, and details.

Solution

• Proposed Solution: The authors propose a comprehensive automated pipeline for generating immersive 3D VR environments from 2D murals. This pipeline minimizes human intervention while adhering to the original artistic style and details.
• Innovations:
  • Integration of multiple Generative AI techniques to automate object extraction, refinement, and overall scene construction.
  • Development of an iterative description calibration system that uses GPT-4 to refine generated content and guide Stable Diffusion in producing images closer to the original mural style.
  • A modular architecture that allows the pipeline to be extended to other types of cultural heritage and scenes.
• Implementation Steps and Key Technologies:
  1. Object Segmentation and Classification: Grounding Dino and Segment Anything Model (SAM) are used to detect and segment key elements in the murals, with CLIP employed to identify object categories.
  2. Object Refinement: Stable Diffusion combined with ControlNet refines images while preserving the original style's colors and line details.
  3. 3D Modeling: TripoAI converts 2D architectural slices into 3D models, adding three-dimensional representations of plants and mountains.
  4. Scene Reconstruction: Objects are arranged in Unity3D based on the original mural layout, resulting in an immersive VR scene.
  5. Background Generation: Stable Diffusion is used to generate ground textures and replace sky backgrounds.

Research Outcomes

• Specific Outcomes:
  • Successfully converted the mural scene of the Buddha Temple from Dunhuang into an immersive VR environment.
  • User experience studies show that AI-generated scenes are comparable to traditional handcrafted scenes in terms of immersion, realism, engagement, and emotional resonance.
  • Experts acknowledged the superior 3D effects of AI-generated scenes and suggested incorporating dynamic elements and aesthetic enhancements in the future.
• Advantages Over Existing Solutions:
  • High scalability and efficiency, enabling rapid reconstruction of large-scale scenes.
  • Superior visual detail and realism for individual objects compared to handcrafted scenes, though overall scene consistency requires improvement.
• Experimental and Evaluation Results:
  • Quantitative analysis indicates no significant differences between AI-generated and handcrafted scenes across major user experience dimensions.
  • Participants and experts highlighted the advantages of AI-generated scenes in model precision, lighting effects, and detail handling, while noting the need to address inconsistent artistic styles and accuracy.
• Limitations and Future Directions:
  • Further improvements are needed in representing historical details, such as specific architectural features (e.g., dougong and caisson structures).
  • Current scene layouts still require manual design to ensure scene completeness.
  • Future work could incorporate dynamic scene elements, enrich interactive experiences, and develop more advanced AI models to enhance automation.
  • The system's scalability allows for personalized customization of mural scenes, presenting opportunities for deeper exploration in future research.

Through the above analysis, the authors' research provides a viable framework for applying Generative AI to the field of cultural heritage preservation, while proposing specific directions for improving the fidelity and consistency of generated content.