PinpointFly: An Egocentric Position-control Drone Interface using Mobile AR

Title of the Paper

PinpointFly: An Egocentric Position-control Drone Interface using Mobile AR

Paper Information

• Subject Area: Human-Computer Interaction, Augmented Reality Technology, and Drone Control Interface Design
• Keywords: Spatial Awareness, Drone Positioning, Augmented Reality, Visual Navigation, Human-Computer Interaction

Research Background and Problem

• Problem or Challenge: Traditional joystick-based drone speed control methods are unsuitable for precise positioning tasks due to limitations such as insufficient spatial awareness, complexity in drone coordinate system control, and lack of precision in speed control mechanisms. These issues particularly affect the performance of novice pilots and increase the learning curve.
• Significance: Accurate positioning and trajectory control of drones are critical for indoor applications such as video shooting, structural inspection, and short-range delivery. In confined indoor environments, drones must operate quickly, flexibly, and safely.
• Motivation and Related Work:
  • Existing AR-based visual overlay technologies improve the intuitiveness of drone flight paths but lack precise positioning capabilities.
  • User-centric control interfaces, such as "headless mode" and body posture control, have been shown to be more efficient than traditional drone coordinate system controls.
  • Professional photography drones currently rely heavily on automated algorithms to generate fixed flight paths, which cannot support real-time interactive drone operation needs.

Solution

• Proposed Method: PinpointFly is a user-centric drone positioning control interface that leverages mobile augmented reality (AR) to allow users to intuitively and directly position and rotate drones in flight.
• Innovations:
  • Enhancing pilots' spatial awareness using virtual projection shadows (orthogonal projections of the drone on the ground), thereby improving the intuitiveness of drone position and orientation.
  • Shifting from traditional drone coordinate system-based controls to user coordinate system-based operational logic.
  • Introducing a position control concept to enable precise drone operation, moving away from traditional speed-based methods.
• Implementation Steps and Key Technologies:
  1. Interaction Design:
     • Introducing a mobile AR view to display the drone's position, allowing users to drag the shadow on the touchscreen to set the drone's position.
     • Using height and orientation sliders to adjust the drone's altitude and rotation angle, respectively.
  2. Four Motion Control Modes:
     • DragFly: Real-time drone positioning by directly dragging to change its direction and position.
     • PointFly: Users preset the position and orientation, and the drone moves to the target point with a single tap of the flight button.
     • TrajectoryFly: Users preset a flight path, and the drone subsequently follows the defined trajectory automatically.
     • WaypointFly: Users can set multiple waypoints, and the drone sequentially visits these points to complete the task.
  3. Technical Implementation:
     • Utilizing an optical capture system (e.g., OptiTrack) and ArUco markers for drone position tracking.
     • Wireless communication to enable data exchange between the mobile AR device, server, and drone.
     • A PID (Proportional-Integral-Derivative) control algorithm to allow the drone to track virtual target positions.

Research Outcomes

• Specific Results:
  • PinpointFly significantly improved drone positioning accuracy in indoor environments. Compared to traditional joystick-based speed control methods, positioning error was reduced by 50%, and task completion time decreased by 33%-45%.
  • User studies demonstrated that this interface is more user-friendly than traditional joysticks, reducing workload for beginners and improving operational efficiency.
• Advantages:
  • Enhanced Precision: Virtual shadows improve users' spatial awareness of the drone's position.
  • Simplified Operation: Users can set altitude and rotation naturally through touchscreen interactions, eliminating the need for complex joystick inputs.
  • Multi-Mode Support: Motion control modes cover needs ranging from real-time operation to predefined path planning.
• Experimental or Evaluation Results:
  • Two user studies validated PinpointFly's superior performance metrics (e.g., time, distance error, workload) in drone positioning and video shooting tasks.
  • Users exhibited different preferences for the four motion modes; for instance, WaypointFly was deemed more suitable for planning tasks, while DragFly was preferred for real-time operations.
• Limitations and Future Directions:
  • The current system relies on an optical capture system for drone position tracking, limiting its application scope.
  • The system does not adequately address issues of shadow occlusion by objects in the scene.
  • Direct comparisons with gesture-based AR drone interfaces have not been conducted.
  • Future research should explore adapting the interface to larger spatial environments with hybrid control methods, while improving dynamic shadow size adjustments and other visual guidance techniques.

Application Scenarios

• Inspection and Fault Diagnosis: Quickly positioning drones at target points for building structure or equipment damage inspection.
• Video Shooting: Supporting dynamic drone video shooting, such as close-up photography or trajectory tracking.
• Short-Range Delivery and Assembly: Delivering tools or assembling components in indoor factory settings using PinpointFly.
• Navigation and Guidance: Using drones equipped with displays to guide workers or visitors.