PoemPalette: Facilitating Poetry Creative Exploration and Foundational Understanding through the Ideorealm Alignment of Paintings and Poems

Paper Title

PoemPalette: Facilitating Poetry Creative Exploration and Foundational Understanding through the Ideorealm Alignment of Paintings and Poems

Publication Info

• Topic area: Human-Computer Interaction (HCI) for poetry and visual art creation
• Keywords: Ideorealm Alignment, Scene Graph, Generative AI, Poetry Visualization, Large Language Models, Creative Exploration, Foundational Understanding, Multimodal Interaction, Cultural Heritage, Human-AI Co-Creation

Background and Problem

• Problem / challenge: Existing tools for poetry and painting often rely on black-box AI models that lack transparency and user control, limiting opportunities for creative exploration and deeper understanding of poetic symbols and their visual representations.
• Significance: Bridging poetry and painting through interactive systems can enhance creative engagement and foundational understanding, especially for novices exploring cultural and artistic traditions.
• Motivation and related work: Prior tools focus on visualization or multimodal interaction but fail to provide structured, interpretable intermediate representations. Semantic alignment between poetic text and visual output remains a challenge, particularly for nuanced cultural and symbolic content.

Solution

• Proposed approach: PoemPalette, an interactive tool that integrates the Ideorealm Alignment of Paintings and Poems (IA-PP) theory with Scene Graphs (SG) and Generative AI to enable users to explore and visualize poetry creatively.
• Novelty:
  1. Development of a structured, interpretable Scene Graph (SG) representation to align poetic symbols with visual elements.
  2. Integration of semantic alignment techniques for controllable text-to-image generation, preserving cultural and symbolic fidelity.
  3. Use of a Large Language Model (LLM) as an intelligent agent to support foundational understanding of poetry through interactive dialogue.
  4. Empirical validation through user studies demonstrating PoemPalette’s effectiveness in enhancing creative exploration and understanding.
• Procedure and key techniques:
  • Extract poetic symbols and relationships using LLM-based semantic parsing to construct SGs.
  • Enable users to interactively modify SGs and generate aligned visual elements using Stable Diffusion XL with LoRA for cultural style adaptation.
  • Provide an LLM-powered agent for contextual explanations and Q&A.
  • Visualize emotional tones and color palettes to deepen engagement with poetic themes.

Results

• Concrete findings:
  • PoemPalette significantly improved creative exploration scores (e.g., Exploration: 82.00 vs. 67.50 for AI baseline; CSI total: 77.59 vs. 66.33 for AI baseline).
  • Foundational understanding scores increased post-intervention (e.g., Chinese poetry group: pretest 43.65 to posttest 79.9).
• Advantage over baselines:
  • Outperformed both AI and non-AI baselines in creativity support (e.g., Enjoyment, Expressiveness) and user experience (e.g., Stimulation, Novelty).
  • Reduced cognitive workload compared to non-AI methods (e.g., lower NASA-TLX Effort scores).
• Experiments / evaluation:
  • Conducted a between-subjects study with 60 participants (30 each for Chinese and Japanese poetry), comparing PoemPalette, AI, and non-AI baselines.
  • Evaluated using standardized instruments (CSI, UEQ, NASA-TLX, SUS) and pretest–posttest comprehension tests.
• Limitations and future work:
  • Challenges in representing abstract or highly rhetorical poetic elements.
  • Limited cultural scope (focused on Chinese Tang poetry and Japanese haiku).
  • Planned improvements include building a symbol repository, enhancing SG flexibility, and conducting longitudinal studies.

Summary

PoemPalette is an interactive system that bridges poetry and painting through the Ideorealm Alignment of Paintings and Poems (IA-PP) framework. By combining Scene Graph-based semantic alignment, generative AI, and LLM-powered assistance, it enables users to explore and visualize poetic symbols creatively. User studies demonstrated its effectiveness in enhancing creative exploration and foundational understanding compared to AI and non-AI baselines. Future work aims to expand cultural adaptability, refine graphical representations, and explore long-term learning effects.