BirdViewAR: Surroundings-aware Remote Drone Piloting Using an Augmented Third-person Perspective

Title of the Paper

BirdViewAR: Surroundings-aware Remote Drone Piloting Using an Augmented Third-person Perspective

Paper Information

• Domain: Augmented Reality (AR) and Remote Drone Piloting Technology
• Keywords: Augmented Reality, Third-person Perspective (TPV), Spatial Awareness, Drone, Remote Flight Control, Automated Flight, User Experience, AR Overlay, Visualization Technology

Research Background and Problem Statement

• Problems and Challenges:

  1. Traditional first-person perspective (FPV) in drone piloting suffers from blind spots, limiting the pilot's spatial awareness of the surrounding environment.
  2. Existing methods (e.g., multi-camera setups, 360-degree cameras, and SLAM technology) face real-time limitations and fail to fully address the blind spot issue.
  3. Static or preset third-person perspectives (TPV) cannot adapt to dynamic flight requirements, and pilots may struggle to accurately perceive spatial states such as drone altitude and direction.

• Significance of the Research:

  • Addressing the issue of blind spots is essential for remote drone control, especially for beyond visual line of sight (BVLOS) operations, ensuring safety and operational efficiency.
  • Drones have extensive applications in disaster investigation, package delivery, remote maintenance, and video production. Enhancing the piloting experience will significantly advance drone-related technologies.

• Evaluation of Related Work:

  • Previous studies have made progress based on TPV but largely lack dynamic adjustment capabilities and fail to adequately address pilots' spatial awareness.
  • AR overlays have shown some effectiveness in improving spatial awareness but are primarily focused on VLOS (within visual line of sight) scenarios and have not been effectively integrated with TPV for BVLOS operations.

Proposed Solution

• Research Methodology:

  • The BirdViewAR system is proposed, utilizing an augmented third-person view (TPV) of the drone combined with AR overlay technology to provide pilots with enhanced spatial awareness and flight status feedback.
  • The system consists of a primary drone (for task execution) and a follower drone (for providing TPV), with optimized algorithms dynamically adjusting the follower drone's position and perspective.
  • AR graphical overlays are integrated into the TPV, including altitude indicators, directional markers, camera field of view (FOV) cones, and distance guidance between the drone and surrounding areas.

• Innovations:

  1. Dynamic TPV Framing: Automatically adjusts the follower drone's altitude, position, and camera orientation based on the primary drone's speed and direction, capturing real-time flight status and orientation.
  2. AR Overlay Design: Enhances visual feedback to improve understanding of the drone's surrounding spatial states, addressing visibility issues in traditional TPV.
  3. A dynamic drone tracking control model based on visual and optimization algorithms is proposed, effectively reducing latency issues associated with GPS dependency.

• Technical Implementation:

  • Two programmable drones (DJI Mavic 2 Pro and Parrot Anafi 4K) are utilized.
  • Communication and computation are achieved via Linux and Windows workstations, including position calculations, AR overlay generation, and follower drone control.
  • Python scripts are employed to implement vision algorithms based on OpenCV and PID (Proportional-Integral-Derivative) control algorithms for managing drone movement.

Research Outcomes

• User Experience Improvement:

  • Experimental validation shows that BirdViewAR significantly enhances novice pilots' spatial awareness and flight control performance.
  • Comparative results indicate that combining AR overlays with dynamic TPV framing improves pilots' understanding of spatial relationships between the primary drone and targets, reducing operational anxiety.

• Experimental Data:

  • AR overlays effectively assisted pilots in the "nose-in-circle" task (maintaining continuous circling around a target), reducing error rates by 68%.
  • Dynamic TPV increased success rates in rapid localization tasks (successfully locating within 5-8 meters of the target), with an improvement of 37%.

• Advantages Analysis:

  • BirdViewAR achieved higher task efficiency and safety compared to FPV or static TPV interfaces, enabling pilots to plan flight paths more accurately and reduce unnecessary movements.

• Limitations and Future Directions:

  • Task scenarios were simple, and BirdViewAR's performance in complex 3D flight environments remains unverified.
  • Environmental factors such as lighting and wind pose challenges to visual positioning capabilities.
  • Follower drones face higher risks in complex environments (e.g., forests), necessitating the integration of more advanced obstacle avoidance mechanisms in the future.
  • Scalability improvements: Incorporating advanced SLAM technology and dynamic adjustment mechanisms.

Conclusion

BirdViewAR enhances spatial awareness for remote drone operations through augmented TPV and AR overlay technology, making it particularly suitable for BVLOS scenarios. While preliminary experiments have yielded positive results, future work will focus on optimizing system performance and expanding applicable scenarios.