Towards LLM-powered Assistive Drone for Blind and Low Vision Users

Paper Title

Towards LLM-powered Assistive Drone for Blind and Low Vision Users

Publication Info

• Topic area: Assistive technology for Blind and Low Vision (BLV) users using drones and Large Language Models (LLMs).
• Keywords: assistive drones, blind and low vision, large language models, natural language interaction, participatory design, human-robot interaction, accessibility, voice-based interfaces, object localization, spatial orientation.

Background and Problem

• Problem / challenge: Current assistive technologies for BLV users, such as wearable devices or robotic canes, are limited in their ability to provide flexible, natural communication and handle complex real-world tasks. Drones, while promising, lack seamless interaction methods tailored to BLV users.
• Significance: Addressing these limitations could significantly improve the independence and quality of life for BLV individuals, enabling them to perform tasks such as object localization, spatial orientation, and navigation in unfamiliar environments.
• Motivation and related work: Prior research has explored assistive robots and wearable devices for BLV users, as well as voice-based interactions and LLM-powered robotics. However, there is limited work on leveraging LLMs to enable natural and flexible communication for assistive drones tailored to BLV users.

Solution

• Proposed approach: Development of an LLM-powered voice-based assistive drone prototype for BLV users, capable of translating natural language commands into drone actions and extracting visual information from images.
• Novelty:
  1. A prototype leveraging LLMs for both command translation and vision-based tasks.
  2. Participatory design approach involving BLV users and domain experts to iteratively refine the prototype.
  3. Exploration of drones as a novel assistive technology form factor for BLV users.
• Procedure and key techniques:
  • Conducted a formative study with 9 BLV users to identify use cases and design considerations.
  • Developed a prototype using a Tello EDU drone, Google Speech-to-Text, and GPT-4o for code generation and vision tasks.
  • Iteratively refined the prototype based on feedback from 3 BLV users and 5 experts.
  • Evaluated the final prototype with 6 BLV participants in a controlled lab setting.

Results

• Concrete findings:
  • Achieved a mean System Usability Scale (SUS) score of 73.3, indicating acceptable usability.
  • Task-specific success ratings: 4.0/5 for object localization, 3.8/5 for object recognition, and 3.2/5 for spatial orientation.
  • AI response accuracy: 85% success rate for user queries, with main errors in code generation (7.04%) and vision-based tasks (6.35%).
  • Average response times: 1.46 seconds for code generation, 3.18 seconds for vision-based tasks.
• Advantage over baselines:
  • Enabled flexible, natural language interaction without requiring predefined commands.
  • Provided hands-free, multi-viewpoint assistance, surpassing the capabilities of wearable devices like smart glasses or mobile apps.
• Experiments / evaluation:
  • Mixed-methods user study with 6 BLV participants, including onboarding, exploration tasks (object localization, recognition, and spatial orientation), and post-test interviews.
  • Data collected included Likert-scale ratings, SUS scores, system logs, and qualitative feedback.
• Limitations and future work:
  • Conducted in controlled indoor settings, limiting generalizability to real-world scenarios.
  • Dependent on network connectivity and off-the-shelf components, which introduced latency and occasional errors.
  • Future work should explore real-world deployments, offline models, improved AI accuracy, and more user-friendly hardware designs.

Summary

This paper presents an LLM-powered voice-based assistive drone prototype for BLV users, designed to translate natural language commands into drone actions and perform vision-based tasks. Through participatory design and iterative development, the prototype demonstrated promising usability and flexibility, addressing unmet needs in object localization, recognition, and spatial orientation. While technical and social challenges remain, such as AI errors and public acceptance of drones, this work provides a foundation for future research into assistive drone technologies for BLV individuals.