EyeSee: Enhancing Art Appreciation through Anthropomorphic Interpretations from Multiple Perspectives

Research Background and Issues

What problems or challenges did the authors identify?

Traditional online art appreciation platforms, while reducing geographical constraints and lowering costs, often lack interactivity, resulting in a passive viewing experience that struggles to foster deep user engagement. Furthermore, how to enhance user participation through simulating roles with different perspectives remains underexplored.

Why is this issue important?

Art appreciation not only promotes personal emotional and intellectual growth but also challenges ethics and values while deepening cultural understanding. However, online experiences in the digital age often lack such depth. Particularly in the context of online art appreciation, improving user interaction is of significant importance.

Research Motivation and Related Work

Generative AI technologies, especially large language models (LLMs), have shown potential in enhancing interactive experiences by simulating human-like roles. For example, existing research indicates that role simulation can improve non-experts' understanding of complex topics and enhance student learning outcomes. However, there is a lack of systematic exploration of using multi-role perspectives to strengthen user interaction in the field of art appreciation.

Solution

What methods or solutions did the authors propose?

The authors designed a system called EyeSee, which provides art interpretation to users from multiple perspectives through three distinct role modes (Narrator, Artist, and In-Situ). They employed two task-based conversations—narration and recommendation—to explore how these roles influence user engagement and satisfaction.

• Narrator Mode: Provides third-person narration, offering objective background and information about the artwork.
• Artist Mode: Interprets the artwork from the artist's perspective, focusing on creation motives and intentions.
• In-Situ Mode: Simulates objects or characters within the artwork, delivering immersive storytelling from an internal perspective.

What are the innovative aspects of this solution?

1. Role Diversity: Generates art interpretations from three perspectives, tailored specifically for art appreciation scenarios.
2. AI Support: Utilizes large language models (LLMs) and multimodal models (e.g., Segment Anything Model) to generate detailed narrations and recommend artworks.
3. User Experience Optimization: Evaluates the system's effectiveness through quantitative experiments analyzing aesthetic appeal, perceived usability, and immersion.

What are the implementation steps and key technologies used?

1. System Design:
   • Three roles narrate information in a humanized style (Narrator, Artist, and In-Situ).
   • Tasks include narration conversations (exploring engagement with role narrations) and recommendation conversations (suggesting artworks based on user interests).
2. Technical Implementation:
   • The backend integrates visual thinking strategies and chain-of-thought prompting to optimize generative AI roles' art knowledge.
   • Pre-trained models (e.g., LLaVA-Docent, multimodal language models) are used to generate context-rich responses and recommendation rationales.
3. Experimental Design:
   • 24 art-interested participants completed tasks related to three paintings of different styles.
   • Comprehensive analysis of participant performance and experiences was conducted using behavioral logs (user actions), questionnaires (user satisfaction), and interview feedback (user experience).

Research Outcomes

What specific outcomes were achieved?

1. Enhanced User Interaction:
   • The In-Situ role achieved the highest overall engagement and appeal, particularly excelling in focused attention, aesthetic appeal, and reward factor.
   • The Artist role scored lowest in usability evaluations due to subjective and inconsistent information.
2. Cross-Task Conversation Results:
   • The In-Situ role maintained the highest user relatability and believability across narration and recommendation tasks, indicating its suitability for sustaining user interest.
   • The Artist role showed improved consistency in recommendation tasks but received lower scores in narration tasks due to subjective content and overly high knowledge expectations.
3. User Feedback Support:
   • Behavioral logs showed that users interacted longer and asked more questions with the In-Situ role, demonstrating high behavioral engagement.
   • Emotionally, the In-Situ role provided a strong sense of time travel and multi-sensory experiences through its personified storytelling, making it highly appealing.
4. Theoretical Expansion:
   • The results validated the "Narrative Transportation Theory," demonstrating that role perspectives can influence users' attitude changes and emotional investment.

What advantages does it have compared to existing solutions?

• Multi-Role Dimension Innovation: Existing multi-language models primarily focus on single-perspective narration, whereas EyeSee integrates three unique perspectives, promoting narrative diversity through role embodiment.
• Immersive Experience Design: Particularly, the In-Situ role enhances emotional and cognitive engagement through personified storytelling, offering both novelty and practicality.
• Digital Art Education: Generative AI deepens the exploration of artworks, addressing the lack of interactivity in traditional online museums.

What are the experimental or evaluation results?

Quantitative analysis of participant interaction records and satisfaction revealed:

• Statistically Significant Differences: The In-Situ mode demonstrated advantages in focused attention, appeal, and response satisfaction (p<0.001).
• Cross-Mode Comparison: The In-Situ role consistently led in scores for consistency and believability across task conversations.

Limitations and Future Directions

1. Sample Bias: Participants were primarily art enthusiasts; future research could include a broader audience, such as general viewers and art experts.
2. Knowledge Generation Limitations: The content generated by the Artist role lacked detail and exhibited "LLM hallucinations." Future work could incorporate domain-specific model fine-tuning for optimization.
3. Dynamic Personalization: The study currently relies on fixed system roles; future exploration could focus on adaptive designs that adjust role performance based on real-time user needs.

Through the EyeSee system, the authors significantly improved the user experience of digital art appreciation and proposed an innovative multi-role narrative method. This provides new directions for future digital cultural experiences and generative AI role research while showcasing pathways to enhance emotional and cognitive engagement.