Escapement: A Tool for Interactive Prototyping with Video via Sensor-Mediated Abstraction of Time

Document Title

Escapement: A Tool for Interactive Prototyping with Video via Sensor-Mediated Abstraction of Time

Document Information

• Subject Area: Human-Computer Interaction Design
• Keywords: Video prototyping, creativity support tools, cross-device interaction, sensor-mediated interaction, time abstraction, multi-screen interaction, design tools, rapid prototyping

Research Background and Problem Statement

• What problems or challenges did the authors identify?

  1. Current video prototypes, while widely used in design, lack interactivity. Designers can only watch them without directly manipulating interactions, resulting in insufficient "experiential" engagement.
  2. Creating highly interactive and visually refined prototypes often requires significant engineering costs, involving extensive coding and development efforts.
  3. Rapid iteration and synchronization of device states are challenging when designing cross-device interactions or multi-screen applications.

• Why is this problem important? Video is a widely adopted design tool, but its static nature limits designers' flexibility in exploring and validating dynamic interactions. Providing methods for highly interactive and rapid prototyping can support designers in deeply exploring and shaping design concepts, ultimately contributing to higher-quality interaction design.

• Research Motivation and Related Work:

  1. The authors were inspired by existing interactive design prototyping methods, such as video prototyping and smoke-and-mirror prototypes, but found these tools insufficient in integrating real-time sensor data and supporting cross-device collaboration.
  2. The time dimension of video is often overlooked due to its lack of interactive control. This research aims to introduce "time" as a key design material.
  3. Previous studies, such as the Astral tool, demonstrated the potential of screen scraping and sensor-driven interactions but did not treat video as a primary design material.

Solution

• What methods or solutions did the authors propose? The authors proposed a novel design tool, Escapement, which establishes a flexible mapping between sensor values and dynamic video playback control, introducing the concept of "video-escapement prototyping." This tool allows designers to rapidly create video-based interactive prototypes on single or multiple device screens.

• What are the innovative aspects of this solution?

  1. Time as a Design Material: Abstracting video time and enabling flexible manipulation of the time dimension through sensor input, making time a designable and interactive material.
  2. Standardized Mapping of Sensors and Video: Normalizing sensor inputs and video timelines to a range of 0 to 1, allowing seamless combination and substitution of different sensors and videos.
  3. Multi-Device Support: Designers can easily synchronize state changes across screens and devices using Escapement, supporting interactions over local networks and cloud-based devices.
  4. Rapid Iteration with Low Technical Barriers: Without requiring complex coding, designers can quickly create interactive designs by dragging, dropping, and selecting sensor mapping options.

• What are the implementation steps? What key technologies were used?

  1. Video Preparation and Slicing: Using the tool to decompose videos into a series of static image frames.
  2. Sensor Mapping Configuration: Defining how sensor values dynamically map to video frames, supporting filters, inversions, and easing functions.
  3. Cross-Screen and Device Synchronization: Sharing sensor data across devices via UDP multicast or Azure cloud services.
  4. Real-Time Adjustment and Multi-Angle Experimentation: Providing a control panel for designers to adjust sensor inputs or animation mappings during the design process.

Research Outcomes

• What specific outcomes were achieved?

  1. Introduced a unique "video-escapement prototyping" method, redefining the role of video in interaction design.
  2. Developed the "Escapement tool," enabling rapid video prototyping with real-time sensor interaction across multiple devices.
  3. The tool has been used by over 20 researchers and designers in various design scenarios, including remote work, tilt-responsive interfaces, and dual-screen devices.

• What advantages does it have compared to existing solutions?

  1. No coding required, yet supports the creation of complex interactive prototypes.
  2. Compatible with different sensors and devices, not limited to a single hardware or platform.
  3. Lowers the barrier for video and interaction design, reducing the time from concept to operational prototype.

• What were the experimental or evaluation results?

  1. Over three years of practical use, the tool demonstrated strong support for rapid design and diverse creative exploration.
  2. Real-world application cases highlighted its flexibility, such as remote device data-sharing responses and quick sensor input switching.
  3. Demonstrations at events like Montreal CHW’23 emphasized the tool's usability and efficient interactivity.

• Limitations and Future Directions:

  1. Limited Support for Discrete Interactions: Currently, it primarily supports continuous time-dimension interactions, with limited support for state-switching scenarios.
  2. Restricted to Visual Feedback: Lacks support for other feedback modalities such as audio and haptics.
  3. Limited Transition from Prototype to Final Code: Sensor logic in prototypes has not yet been implemented for code export or further development reuse.
  4. Future Directions:
     • Support for non-visual modalities, such as vibration feedback.
     • Enhanced integration with running code or actual software systems.
     • Optimization of latency issues in cloud-based synchronization.