Jigsaw: Authoring Immersive Storytelling Experiences with Augmented Reality and Internet of Things

Document Title

Jigsaw: Authoring Immersive Storytelling Experiences with Augmented Reality and Internet of Things

Document Information

• Subject Area: Design and creation of immersive storytelling experiences combining augmented reality (AR) and the Internet of Things (IoT)
• Keywords: storytelling, augmented reality (AR), Internet of Things (IoT), authoring tools, immersive experiences, user engagement, multisensory interaction, interaction design, digital storytelling, creative tools

Research Background and Problem

• Problem or Challenge:

  • Current AR-based immersion is primarily limited to visual presentation, constrained to 3D images on screens.
  • Creating multisensory immersive storytelling experiences typically requires high costs, complex technologies, and specialized equipment, making it inaccessible to general users.
  • Few systems attempt to combine virtual content ("pixels") with physical augmentation ("atoms"), and existing tools often demand high levels of expertise, unsuitable for non-technical users.

• Significance:

  • Immersive storytelling enhances emotional expression and user engagement, holding significant application value in education, entertainment, and social contexts.
  • Integrating AR with IoT can enable more accessible and low-cost immersive experiences supported by mobile devices and everyday smart home facilities.

• Research Motivation and Related Work:

  • Inspired by classic interactive experiences (e.g., 4D cinemas and participatory theater), the research team aims to develop a user-friendly storytelling system by coordinating mobile AR and IoT technologies.
  • Previous studies have focused on single types of augmented devices (e.g., projection, VR, or IoT) but have yet to integrate virtual and physical augmentation to support comprehensive immersive storytelling.

Solution

• Proposed System:

  • Jigsaw: A system that allows general users to experience and create multisensory immersive stories using mobile AR and off-the-shelf IoT devices.

• Innovations:

  1. Virtual-Physical Fusion: Overcomes the limitations of visual-only augmentation by integrating IoT physical devices (e.g., smart lighting, fans, and speakers) to create multisensory storytelling experiences blending "pixels and atoms."
  2. Simplified Authoring: Designed for non-technical users, offering a slide-like scene sequence editing tool that enables users to define complex story sequences through keyword triggers and touch interactions.
  3. Multi-Role Interaction: Assigns different roles (narrator, actor, audience) to multiple users, enabling personalized immersive experiences.

• Implementation Steps and Key Technologies:

  1. Immersive Story Design:
     • The system creates storytelling experiences scene by scene, combining AR and IoT artistic effects and trigger behaviors.
     • Example stories include Benjamin Franklin's kite experiment, the fable "The North Wind and the Sun," and the children's story Goodnight Moon.
  2. User Authoring Tools:
     • Enables the definition of "behaviors" (states of virtual objects or IoT devices), "scenes" (individual story segments), and "triggers" (controls for transitioning from the current scene to the next, such as keyword recognition, touch, or screen taps).
     • Provides an intuitive cockpit-style interface, allowing users to adjust objects' relative positions, behaviors, and effects directly within AR scenes.
  3. Implementation Details:
     • Utilizes existing development tools like Lens Studio for AR tracking, multi-user collaboration, and basic audio control.
     • Interacts with smart home devices via the Kasa smart device API and Bluetooth modules.

Research Outcomes

• Specific Achievements:

  1. Designed and implemented the Jigsaw immersive storytelling system and authoring tools.
  2. Created three representative immersive stories, demonstrating the potential of combining virtual and physical augmentation.
  3. Conducted user studies to validate Jigsaw's usability and user-friendliness.

• Advantages Over Existing Solutions:

  • Achieves multisensory immersion using generic IoT devices (rather than custom hardware or expensive equipment).
  • Lowers the barrier to creation, enabling non-technical users to build complex interactive stories.
  • Enhances the synergy between virtual visual augmentation and physical environmental effects, delivering a more comprehensive sense of immersion.

• Experiment/Evaluation Results:

  • Immersive Experience: 20 participants from technology companies (non-expert users) were divided into 10 groups and found Jigsaw to provide more engaging, interactive, and memorable storytelling experiences.
  • User Creation Feedback: Despite limitations with some complex behaviors (e.g., setting specific animation effects), all participants successfully completed storytelling tasks and praised the tool's ease of use and flexibility.
  • Challenges:
    • Better coordination of multisensory inputs is needed to reduce sensory overload during participatory experiences.
    • Long-form stories face challenges related to navigation complexity and collaborative design of repetitive elements.

• Limitations and Future Directions:

  1. Currently supported device types are limited to smart lighting, fans, and speakers. Future work should explore additional devices such as robots and projectors.
  2. User studies primarily focused on adult technical users; future research should consider the experience of children, an important target audience for storytelling.
  3. Scenario design for complex actions and animations requires further optimization, potentially leveraging AI technologies for context generation.