Datamancer: Bimanual Gesture Interaction in Multi-Display Ubiquitous Analytics Environments

Research Background and Problem Statement

• Challenges and Issues Identified by the Authors:

  1. Existing data visualization interactions are primarily limited to small-scale single devices, such as personal computers, or require pre-installed bulky equipment.
  2. Gesture interaction has not been widely applied in multi-display environments, especially in panoramic multi-display environments for data analysis tasks.
  3. Current mobile and wearable devices often require pre-mapped spatial configurations, lacking the capability for seamless interaction anytime and anywhere.
  4. There is a lack of in-depth research on interaction design that combines body and gesture control in cross-device environments.

• Significance:

  • As data visualization scenarios transition from single desktop environments to multi-display and augmented reality multi-device environments, new interaction paradigms are becoming essential.
  • In scenarios such as data analysis, collaborative decision-making, and presentations, the ability to interact flexibly and efficiently across multiple displays will further enhance work efficiency and user satisfaction.

• Research Motivation and Related Work:

  • This study builds on the historical development of video analysis systems, wearable devices, and bimanual interaction technologies, integrating multi-disciplinary research experiences from "gesture and voice multimodal interaction" to "display acquisition in distributed analysis environments."
  • The authors reference the concepts of Ubiquitous Analytics and Immersive Analytics, emphasizing seamless integration with data environments.

Proposed Solution

• Proposed Solution:

  1. Datamancer Device: A wearable device prototype that integrates gesture operations and hardware, enabling seamless data interaction across multiple displays.
  2. Three Interaction Modes:
     • Screen Acquisition: Users point and select displays using hand gestures.
     • Mode Switching: Users choose different interaction modes (e.g., zoom, drag).
     • Interaction Execution: Enables various functions such as map navigation, zooming, and data filtering.

• Innovations:

  1. Proposed a lightweight wearable device design combining a chest-mounted gesture tracker and a ring camera, overcoming the limitations of traditional desktop or fixed display devices.
  2. Developed a fully browser-based collaborative environment, allowing dynamic selection of interaction targets across multiple displays.
  3. Supports natural dual-hand six degrees of freedom (6DoF) interaction, offering more flexibility compared to traditional interaction devices.

• Implementation Steps and Key Technologies:

  1. Hardware Design:
     • A miniature camera worn on the hand for display recognition.
     • A Leap Motion gesture tracker mounted on the chest for recognizing dual-hand gestures.
     • Raspberry Pi for computational support, combined with a portable power source for extended use (~10 hours).
  2. Software Architecture:
     • Based on the MyWebstrates platform, enabling cross-platform visualization through a browser.
     • Data synchronization utilizes Automerge's distributed network layer structure, effectively achieving real-time communication between devices.
  3. Interaction Design:
     • Developed a set of common gestures to support operations in data interaction (e.g., selection, dragging, zooming, filtering).
     • Used the ArUco algorithm for screen marking and recognition to enable screen acquisition.

Research Outcomes

• Specific Outcomes:

  1. Device Prototype: Successfully developed the Datamancer prototype device, supporting cross-screen interaction in various scenarios such as personal office work, meeting presentations, and collaborative analysis.
  2. Experimental Validation: Validated the practical application potential of the system through three scenarios (personal office work, meetings, collaborative analysis).
  3. User Study Results:
     • Data experts highlighted its potential use in collaborative analysis, such as traffic management and multi-user collaboration.
     • User testing (12 participants) showed that most users found the system intuitive and easy to use, with a System Usability Scale (SUS) score of 78.125 (above average).

• Advantages Compared to Existing Solutions:

  1. Does not rely on cumbersome head-mounted devices or large screen installations, offering a flexible, wearable solution.
  2. Supports instant interaction and dynamic multi-device operations, enabling quick deployment in different environments.
  3. Achieves natural dual-hand multi-tasking interaction, which is more intuitive compared to mice and XR controllers.

• Experimental or Evaluation Results:

  1. Users quickly adapted to the system and completed distributed multi-screen visualization tasks.
  2. Data experts noted that the system could significantly enhance efficiency during presentations and collaboration.
  3. Application scenarios ranged from classroom teaching to multi-user collaboration and on-site analysis.

• Limitations and Future Directions:

  1. Learning Curve: Initial learning is required for gesture operations, and some users may forget gestures.
  2. Technical Limitations: The current prototype is not fully wireless, imposing some constraints during actual operation.
  3. Design Optimization:
     • Enhance the robustness of gesture recognition and add more user-defined gestures.
     • Introduce visual guidance and real-time feedback mechanisms to improve gesture discoverability.
  4. Expansion Directions:
     • Support simultaneous multi-user usage while minimizing interference.
     • Improve compatibility beyond browser environments to support more professional applications (e.g., local analysis software).
     • Explore new extended input methods that do not rely on large spatial gestures, such as waist-mounted sensor inputs.