InteRecon: Towards Reconstructing Interactivity of Personal Memorable Items in Mixed Reality

Research Background and Issues

• Issues and Challenges: Current digitization methods (e.g., photos, videos, 3D scanning) primarily focus on replicating the appearance of objects but lack the preservation of their interactivity. As a result, digitized objects fail to capture the rich interactive characteristics associated with memories.
• Significance: Physical objects carry rich emotional memories and stories through their interactions and functional use. These interactive qualities are central to enhancing memory preservation and everyday practicality. However, such interactivity is often overlooked or lost in traditional digitization methods.
• Research Motivation: Through formative user studies, the authors found that the interactive characteristics of memory objects (e.g., toys, electronic devices), such as geometric shapes, interface features, and embedded content, are crucial memory triggers. This study aims to address this gap in existing digital preservation efforts by unifying interactivity and memory in digitization.

Solution

• Methods and Techniques: The concept of Interactive Digital Items (IDIs) is proposed, along with the development of an AR prototype tool, InteRecon, which enables users to create and showcase interactive digital objects in an augmented reality environment.
• Innovations:
  1. Proposed design goals for IDIs, emphasizing that digitized objects should retain real-world geometric properties, interactive interfaces, and embedded content.
  2. Introduced a user-friendly creation process to support non-expert users, enabling them to create digital objects with movable joints or functional buttons on electronic devices.
  3. Developed four core functional modules to support interactivity reconstruction: 3D appearance reconstruction, physical transformation addition, interface reconstruction, and embedded content addition.
• Implementation Steps:
  1. 3D Appearance Reconstruction: Create digital models that faithfully replicate the physical appearance of objects using 3D scanning.
  2. Adding Physical Transformations: Allow users to segment models and define physical constraints (e.g., rotation, hinge, or ball joints) to simulate the physical dynamics of objects.
  3. Interface Reconstruction: Enable users to add interactive components such as virtual buttons, knobs, and screens to the digital model and bind corresponding functionalities.
  4. Embedded Content Addition: Allow users to embed songs, photos, videos, or other digital content into the model.

Research Outcomes

• Specific Results:

  1. The concept of IDIs was well-received by users and evaluated as a more expressive form of memory preservation.
  2. Using InteRecon, users successfully reconstructed interactive digital objects, such as adding joint movements to toy dogs or button functionalities to electronic devices.
  3. IDIs were considered to have stronger emotional connections compared to traditional static 3D scans or photos.

• Advantages:

  • Compared to existing solutions, IDIs not only focus on the static appearance of objects but also preserve their interactivity, enabling the constructed digital objects to showcase multidimensional memory representations. For instance, AR models reconstructed by users can achieve dynamic interactions consistent with real-world objects.
  • Rich customization features allow non-expert users to easily engage in interaction design, simplifying complex functionalities into user-friendly editing operations.

• Experiment and Evaluation Results:

  1. Functionality Evaluation: Users provided overall positive feedback on the functionality modules (e.g., ease-of-use score: 5.81/7, expressiveness score: 6.08/7).
  2. Successful Cases: Users reconstructed various objects using InteRecon, including toys with realistic physical joints, electronic devices with functional buttons, and dynamic screens.
  3. Limitations and User Feedback:
     • Model segmentation sometimes triggered users' emotional discomfort (e.g., segmenting a toy's body into different parts).
     • Current physical simulation features are basic and lack material property modeling (e.g., softness, texture).
     • Some operations lacked precision when using AR headsets, such as gesture recognition accuracy.

• Future Directions:

  1. Technical Enhancements: Improve material and physical property simulations to support more complex joints and dynamics, such as pulley systems or friction.
  2. Emotional Optimization: Explore transitions between 2D interfaces and 3D environments to reduce the emotional impact of highly realistic segmentation.
  3. Exploration of New Application Scenarios: Beyond personal memory preservation, expand to fields such as education, museum exhibitions, and medical simulations, particularly for the vivid presentation of cultural heritage.
  4. Community Building: Create a platform for users to upload and share interactive 3D assets, promoting the adoption of interactive virtual assets.

Conclusion

IDIs provide a new perspective on enriching memory preservation and interaction design, making them widely applicable to both personal and professional contexts. InteRecon effectively integrates interactivity into the 3D reconstruction process, but future efforts should focus on enhancing functionality diversity and optimizing user emotional experiences.