On the Role of Materials Experience for Novel Interactions with Digital Representations of Historical Pop-up and Movable Books

Title of the Paper

Exploring the Role of Material Experience in the Digital Representation and Interaction of Historical Pop-up Books

Bibliographic Information

• Domain: Human-Computer Interaction (HCI), Cultural Heritage Digitization (with a focus on historical pop-up and movable books)
• Keywords: Materiality, Material Experience, Cultural Heritage, Books, Mixed Reality, Virtual Reality

Research Background and Issues

• Identified Problems/Challenges:

  1. Due to the fragility and preciousness of cultural heritage (CH) artifacts, users are often unable to directly touch these objects, leading to a lack of material perception experience.
  2. The digitization of cultural heritage (especially 3D digital representations) is rapidly advancing, but there is a risk of information loss in capturing and conveying the physical properties of artifacts.
  3. Traditional digital experiences of cultural heritage are often designed with a "technology-driven" or "as realistic as possible" approach, without fully exploring the potential of material properties and experiences.

• Why It Matters:

  • Materiality and material experience directly influence people's understanding, emotions, and behaviors toward artifacts.
  • The diversity of materials and their potential for interaction with users is not limited to simulation or technology-driven approaches, opening up new design spaces for creating more engaging and meaningful experiences.

• Research Motivation and Related Work:

  • Although some HCI case studies in cultural heritage have attempted to incorporate the physical properties of artifacts into interaction design, there is little exploration of how to systematically translate the material properties of artifacts into interaction design.
  • This study aims to propose a systematic analysis of the material properties of cultural heritage artifacts based on the framework of material experience and to inspire novel digital interaction forms through these properties.

Solution

• Proposed Methods/Solutions:

  1. Propose a method for characterizing material experience, mapping the material properties of artifacts to digital interaction design.
  2. Use historical movable and pop-up books as case studies to explore how material properties (e.g., foldability, slidability, tearability, ageability, and printability) can inspire new forms of interaction.
  3. Develop two XR-based prototypes (one in mixed reality and one in virtual reality) that leverage material experience to inspire interaction design.

• Innovative Aspects of the Solution:

  • Challenges the traditional design paradigms of "as realistic as possible" and "technology-driven" digital experiences in cultural heritage by proposing an interaction design framework centered on materiality.
  • Expands the application scenarios of material experience from material research to the digital interaction domain of cultural heritage.

• Implementation Steps/Key Technologies:

  1. User Research: Observe and interview users interacting with historical movable books, analyzing these experiences using the material experience framework (sensory, interpretive, emotional, and behavioral).
  2. Feature Abstraction:
     • Extract five key material properties of historical movable books: foldability, slidability, tearability, ageability, and printability.
     • Construct experience diagrams for each property to organize users' perceptions and behaviors.
  3. Prototype Design:
     • Mixed Reality Prototype: Combine head-mounted devices with physical books to enhance material experiences such as opening/closing effects and aging effects.
     • Virtual Reality Prototype: Amplify dynamic scenes and enrich interaction methods to evoke user immersion and engagement in the reading experience.
  4. Technical Implementation: Use 3D modeling, tracking systems (e.g., SteamVR™ 2.0), and engines (e.g., Unity, Unreal Engine) to develop prototypes.

Research Outcomes

• Specific Results:

  1. Five Key Material Properties:
     • Foldability: Opening objects to form 3D dynamic scenes evokes positive emotions such as surprise and anticipation.
     • Slidability: Sliding paper components generates interactive effects, eliciting playfulness and curiosity.
     • Tearability: The fragility of paper encourages careful handling, which can be used as an innovative interaction point in digital contexts.
     • Ageability: Includes features like color fading and wear marks, conveying the historical essence of artifacts.
     • Printability: Detailed printing (colors/textures) guides users' observation and reading behaviors.
  2. Two XR prototypes integrating material experience properties demonstrate how to expand and enhance user interaction with cultural heritage artifacts.
  3. User studies show that, compared to virtual books, users prefer to derive intuitive material experiences from physical books, though XR can enhance overlooked properties.

• Contributions and Advantages:

  • Provides a framework for designing cultural heritage interactions through material experience.
  • Guides designers to focus on the material essence of artifacts rather than relying solely on technological simulation.

• Experimental Results:

  • Experiments show that interaction designs inspired by material experience (e.g., incorporating color changes and aging simulations) enhance user immersion and enjoyment.
  • Users have a strong perception of differences in material properties between virtual and physical artifacts, especially in terms of touch and feedback.

• Limitations and Future Directions:

  1. The artifacts used in this study are mostly contemporary replicas, which may affect the complete restoration of the material experience of historical artifacts.
  2. The study does not deeply explore social interactions (e.g., collaboration and educational functions between children and adults in exhibition contexts).
  3. Technical limitations remain (e.g., accuracy of gesture tracking in XR, real-time synchronization between physical and virtual elements).
  4. Future work should validate this framework across a broader range of artifact types and iteratively evaluate its educational and experiential effects.