AirConstellations: In-Air Device Formations for Cross-Device Interaction via Multiple Spatially-Aware Armatures

Literature Title

AirConstellations: In-Air Device Formations for Cross-Device Interaction via Multiple Spatially-Aware Armatures

Literature Information

• Domain: Human-Computer Interaction (HCI)
• Keywords: Cross-device computing, device arrangement, semi-fixed workspace, multi-device interaction techniques, platform, cross-device tracking, SurfaceFleet, device ecosystem

Research Background and Issues

• Background: As users increasingly utilize multiple devices simultaneously in their personal and professional lives, cross-device interaction techniques have become essential tools for supporting complex work scenarios. However, existing cross-device methods are often based on fixed device setups (e.g., embedded interactive surfaces in environments) or mobile device coordination, both of which face significant limitations in spatial reconfiguration and sensor tracking.
• Challenges:
  • Fixed device configurations struggle to adapt to rapid changes in tasks, users, and contexts.
  • Mobile device ecosystems (e.g., smartphones and tablets) offer limited spatial arrangement options, as devices are typically handheld or placed on flat surfaces.
  • The lack of reliable solutions for tracking device positions and orientations in space restricts the application of many cross-device interaction techniques.
• Motivation:
  • To provide a solution that maintains the modularity and reconfigurability of mobile devices while enabling physical adjustments to device configurations based on user needs.

Solution

• Core Method and Platform:
  • Introduce a system called "AirConstellations" to enable dynamic "in-air" device arrangements.
  • Integrate low-cost, durable sensors into standardized mechanical arms or mounts to achieve position and orientation tracking in three-dimensional space (7DoF model).
  • Provide a "semi-fixed" type of cross-device interaction setup, allowing devices to move freely in space and be positioned in real time via mounts.
• Innovations:
  • Dynamic "in-air" device arrangements: Device positions, orientations, and combinations can be dynamically adjusted to suit user tasks and contextual needs.
  • Sensor-driven interaction behaviors in a user-friendly manner, such as triggering dynamic interface adjustments through close-range device sensing.
  • Propose and define a design space encompassing physical and spatial configurations of devices and their interaction dimensions, totaling 16 core dimensions.
• Implementation Steps and Techniques:
  1. Hardware Design: Utilize universal three-joint mechanical mounts equipped with inertial measurement units (IMU), rotary sensors, and low-cost microcontrollers to support posture sensing.
  2. Sensing and Data Fusion: Combine posture data from IMUs with rotary sensor data to construct stable 3DoF/6DoF device motion models.
  3. Real-Time Software Platform: Provide software tools for data filtering, interaction design, and animation visualization, enabling the system to respond to changes in device position and orientation.
  4. Interaction Behavior Design: Design corresponding feedback and functions for user behaviors based on physical arrangements, such as device proximity, movement, and rotation.

Research Outcomes

• Specific Outcomes:
  1. AirConstellations System: Provides a solution for "semi-fixed" cross-device interaction.
  2. Design Space: Defines 16 core dimensions of dynamic device arrangements to guide hardware and software design.
  3. Application Scenarios and Interaction Techniques:
     • Video conferencing: Offers separate "task space" and "personal space," supporting dynamic switching and personalized interaction.
     • Productivity applications: Triggers toolbar switching or keyboard input redirection when devices are brought closer.
     • Data analysis workspace: Supports horizontal or vertical device stitching, enabling easy multi-monitor extension and personal view sharing.
     • System-level device coordination: Includes cross-device window management and lightweight application migration.
  4. User Interview Feedback:
     • Users highly praised the system's "mobile yet stable" characteristics, emphasizing its flexibility in shared space adjustments.
     • Highlighted the naturalness and smoothness of interaction feedback animations.
     • Suggested further simplification and optimization of unnecessary automated behaviors.
• Advantages:
  • Compared to existing solutions, AirConstellations significantly improves dynamic configuration, modular flexibility, and sensing accuracy.
  • Its high adaptability makes it highly practical in rapidly changing work scenarios.
• Limitations:
  • The current device mount design involves trade-offs between cost and stability.
  • Some users expressed concerns about accidental interactions caused by unintentional operations.
• Future Directions:
  1. Explore device arrangement layouts for group collaboration, supporting multi-user, shared device interaction modes.
  2. Combine other form factors, such as adjustable-height mechanical arms and lighter sensor devices.
  3. Investigate the potential of dynamic device surfaces in virtual reality and augmented reality applications.