CoCapture: Effectively Communicating UI Behaviors on Existing Websites by Demonstrating and Remixing

Document Title

CoCapture: Effectively Communicating UI Behaviors on Existing Websites by Demonstrating and Remixing

Document Information

• Field of Study: Human-Computer Interaction (HCI), User Interface Design, and Prototyping
• Keywords: Rapid UI Prototyping, UI Behavior Communication, Dynamic User Interfaces, DOM Structure Recording and Reproduction, Animation Prototyping, Visual References

Research Background and Problem

• Identified Problems or Challenges:

  1. In the later stages of UI development, static screenshots and sketches fail to accurately convey complex interactions and dynamic behaviors.
  2. Prototyping complex UI behaviors requires significant effort, and existing tools lack support for directly importing assets or behaviors from existing interfaces.
  3. Tools for modifying existing behaviors are scarce, making it difficult to blend high-fidelity and low-fidelity prototypes, thus hindering effective communication.

• Significance:

  • UI behaviors are critical to user experience. However, due to the dynamic nature of interactions, designers and developers often face communication barriers, leading to inefficiencies or misunderstandings.

• Research Motivation and Related Work:

  • Existing tools like SILK and DENIM focus on the early stages of UI design but do not support later-stage dynamic behavior editing.
  • Commercial tools like Figma and Adobe XD require users to build interfaces from scratch, a time-consuming and less scalable process.
  • Literature highlights the need for tools that can import existing asset structures and create dynamic behaviors, while also improving the way behaviors are communicated to enhance understanding and collaboration.

Solution

• Proposed Method or Solution:

  • Introduced an interactive system called "CoCapture," which combines DOM-based recording technology with a demonstrate-remix-reproduce approach:
    1. Captures existing UI behaviors and generates a base scenario.
    2. Allows users to demonstrate new behaviors by directly manipulating interface elements.
    3. Supports subsequent modifications and blending of behaviors.
    4. Integrates natural language descriptions with hyperlinks to animations or elements for clear communication.

• Innovations:

  1. A prototyping tool for dynamic UI behaviors that supports direct manipulation of elements on existing websites.
  2. Prioritizes dynamic UI behaviors as "animation objects" for high-fidelity expression.
  3. Integrates natural language descriptions with visual content through interactive hyperlinks.

• Implementation Steps and Key Techniques:

  1. Behavior Recording: Uses the MutationObserver API to record user interactions and DOM structure changes.
  2. Behavior Editing: Supports real-time adjustments to properties (e.g., width, height, opacity) and demonstrates element movements via drag-and-drop.
  3. Animation Blending: Enables animation editing through timeline sliders and operation settings (clone, delete, etc.).
  4. Enhanced Communication: Links natural language descriptions to animations via hyperlinks and provides concise "animation previews" and "keyframe displays."

Research Outcomes

• Specific Results:

  • CoCapture significantly improved the accuracy of UI behavior descriptions (from 54% to 92%).
  • Reduced the word count of textual descriptions (average 115.71 words reduced to 38.23 words), allowing designers to focus more on expressing dynamic content.
  • Anchored visual references (hyperlinks) greatly enhanced navigation during communication, reducing cognitive load.

• Comparison with Existing Solutions:

  • Compared to traditional tools (e.g., Email + Google Drawings), CoCapture reduced application switching and provided more concise and behavior-specific descriptions.
  • Its ability to communicate dynamic behaviors far surpassed static references like screenshots or sketches.

• Experimental or Evaluation Results:

  • Experiments showed that designers using CoCapture could generate accurate problem descriptions faster and required less response time (average 482 seconds vs. 645 seconds).
  • Users evaluating the problems found CoCapture-generated descriptions clearer and easier to understand (average score 4.21 vs. 1.40).

• Limitations and Future Directions:

  • CoCapture is currently limited to web-based UIs and does not support other platforms (e.g., mobile apps).
  • For simple interface changes (e.g., changing button background color) or low-complexity interfaces, CoCapture may be unnecessary.
  • The system needs further support for basic functionalities like auto-save and hotkey-based undo, as well as displaying user inputs (e.g., clicks) to enhance the user experience.

Conclusion and Outlook

CoCapture significantly improves the efficiency and accuracy of communicating UI behaviors through innovative animation prototyping and hypertext referencing methods, addressing the challenge of expressing dynamic behaviors between designers and developers. Future work could extend its application to mobile apps and other interface scenarios, providing broader tool support and optimizing operational modules throughout the UI design lifecycle.