Sensitive Pictures: Emotional Interpretation in the Museum

Title of the Paper

Sensitive Pictures: Emotional Interpretation in the Museum

Bibliographic Information

• Domain: Human-Computer Interaction (HCI), Affective Computing, Museum Design
• Authors: Steve Benford et al.
• Conference: CHI '22 (ACM Human Factors in Computing Systems)
• Keywords: Affective Computing, Emotional Interaction, Emotion, Experience Design, Museum, Data Souvenirs, Video Analysis, Ambiguity

Research Background and Issues

• Issues and Challenges:
  • Emotion and its impact on human-computer interaction have become significant topics in HCI, but the relationship between emotional interaction and affective computing remains unclear.
  • Museums are exploring how to engage visitors through emotion-driven approaches, but designing such experiences remains a challenge.
• Significance:
  • Many museums are transitioning from traditional information provision to more personalized and emotional visitor experiences. Emotion-driven exhibits can enhance audience engagement and understanding.
  • The integration of affective computing and emotional interaction can provide new perspectives for addressing emotional evaluation and response in the HCI field.
• Motivation:
  • To explore how to design digital technology-based emotional museum experiences to engage the public and enhance understanding of emotion and experience design.
• Related Work:
  • Research spans the fields of affective computing (measuring emotional states through physiological signals) and emotional interaction (focusing on the complexity of emotions and their social construction).
  • In the museum domain, increasing research explores how interactive technologies can create emotional experiences, but the effectiveness and acceptance of such approaches remain to be validated.

Solution

• Core Method and Process:
  • Proposes an emotional visitor experience called "Sensitive Pictures," which achieves emotional triggers and reflection through four steps:
    1. Onboarding: Visitors use a mobile app to select an emotion and trigger an emotional story scenario associated with an artwork.
    2. Self-reporting: Visitors report their personal emotions via the mobile app or video recording.
    3. Simulated “Phone Call”: Visitors interact with a simulated "artist persona," while the system captures facial emotion signals to enhance the experience.
    4. Emotional Souvenir Postcard: Visitors' emotional data is generated and printed as personalized postcards.
• Innovations:
  • Combines "affective computing" and "emotional interaction" to provide visitors with a deeply emotional narrative experience, enhanced by ambiguity in design to encourage data reflection.
  • Utilizes computer vision technology to automatically detect visitors' facial action units (FAUs) for emotion estimation.
  • Designs and provides emotion-intelligent "data souvenirs" to enhance visitors' reflection after the emotional experience.
• Implementation Steps and Technology:
  • Collaborated with the Edvard Munch Museum in Norway to design an integrated experience, including storytelling, audio, and video capture.
  • Used a mobile web app to generate and record visitors' self-reported emotional data.
  • Employed the "BlueMax" computer vision tool to analyze visitors' facial emotional responses.
  • Reflected emotional data through postcards provided to visitors after the experience.

Research Findings

• Specific Results:
  • The study involved 132 public participants, with 39 undergoing in-depth interviews.
  • Most participants reported the experience as impressive and emotionally engaging, with some describing it as "introspective" or "therapeutic."
  • The postcard design and narrative integration successfully rendered emotional data meaningful.
• Advantages:
  • Compared to traditional museum experiences, this design enhances the depth and interactivity of emotional content.
  • Creatively integrates affective computing with visitor interpretation, encouraging further emotional expression beyond the screen.
• Experiments and Evaluation:
  • Self-reports and computer detection revealed certain limitations (e.g., visitors found the emotion slider input confusing, and the accuracy of emotion recognition via facial detection was limited).
  • Despite some constraints in the open-ended nature of background dialogues, visitors found the emotional visualization design insightful.
• Limitations and Future Directions:
  • Limitations: The accuracy of capturing visitors' emotions using self-report scales and slider models needs improvement; there is ongoing debate about whether visitors trust the scientific reliability of the system's emotional outputs.
  • Future Directions:
    1. Explore more diverse emotional visualization models (e.g., Ekman's six basic emotions theory).
    2. Integrate more precise biosensing technologies to improve emotion capture.
    3. Expand research to museum visitor experiences in different cultural contexts.
    4. Consider deeper integration of emotional experiences with technologies such as augmented reality.

Conclusion and Insights

• Overall Evaluation:
  • "Sensitive Pictures" demonstrates how interactive experience design can evoke visitor emotions and highlights the potential of incorporating emotional touchpoints into digital museum experiences.
  • The combination of emotional interaction and affective computing models fosters cross-disciplinary understanding, revealing cultural and technological intersections within contemporary HCI frameworks.