FetchAid: Making Parcel Lockers More Accessible to Blind and Low Vision People With Deep-learning Enhanced Touchscreen Guidance, Error-Recovery Mechanism, and AR-based Search Support

Title of the Paper

FetchAid: Making Parcel Lockers More Accessible to Blind and Low Vision People With Deep-learning Enhanced Touchscreen Guidance, Error-Recovery Mechanism, and AR-based Search Support

Paper Information

• Field of Study: Assistive Technology and Human-Computer Interaction
• Keywords: Parcel delivery, blind and low vision people, accessibility, mobile devices, object detection, computer vision, augmented reality, assistive technology, touchscreen guidance, error recovery mechanism

Research Background and Problem

• Identified Issues or Challenges:

  1. Parcel lockers are becoming increasingly popular in last-mile logistics, but their current designs face significant accessibility challenges, particularly in touchscreen interaction, locating open compartments, and safe navigation.
  2. Existing solutions (e.g., voice prompts or Braille keyboards) are either costly or ineffective for most blind and low vision users, and current research fails to provide comprehensive accessibility support in dynamic, multi-stage scenarios.

• Significance:

  1. With the growing demand for e-commerce and parcel delivery, improving the usability of parcel lockers for blind and low vision individuals holds significant value in terms of social equity and efficient resource utilization.
  2. Technological innovation can greatly enhance the parcel retrieval experience for blind and low vision users in public facilities, addressing real-life needs.

• Research Motivation and Related Work:

  1. Existing methods, such as computer vision-based or crowdsourced assistive tools and wearable object recognition technologies, face limitations such as high hardware requirements, low environmental adaptability, or lack of real-time error recovery capabilities.
  2. This study aims to provide a systematic, integrated design that offers a solution with interactive guidance and precise navigation capabilities without relying on additional hardware.

Solution

• Proposed Method: Developed a standalone smartphone application, FetchAid, which assists blind and low vision users in retrieving parcels from lockers through deep learning-enhanced real-time touchscreen interaction guidance, an error recovery mechanism, and AR-based search support.

• Innovations:

  1. Deep learning-based touchscreen finger and button detection, providing real-time voice guidance and error recovery mechanisms.
  2. Use of augmented reality technology with planar ray detection and visual-inertial odometry to align the user's spatial position with the target locker and provide voice navigation.
  3. Automated QR code scanning functionality to simplify user input and operational steps.
  4. Specially designed touchscreen interaction feedback and navigation safety features to prevent collisions and accidental locker door closures.

• Implementation Steps and Key Technologies:

  1. Touchscreen Interaction Phase: Using deep learning models to detect the touchscreen and user fingers, providing real-time voice feedback to guide users to press the correct buttons, along with error detection and recovery mechanisms.
  2. Locker Search Phase: Using OCR to identify locker location information on the screen and guiding users to the open locker through AR navigation while avoiding potential collisions.
  3. System Integration and Deployment: Implementing real-time user localization using the phone's ARKit and optimizing the object detection network with TensorFlow Lite for efficient operation on mobile devices.

Research Outcomes

• Specific Results:

  1. FetchAid significantly improved the success rate of blind and low vision users in operating parcel locker touchscreens and locating open locker doors.
  2. Experiments showed a notable reduction in recovery time and psychological stress during error operations, enhancing the user experience.

• Advantages Over Existing Solutions:

  1. Does not rely on additional wearable devices or expensive hardware.
  2. Provides comprehensive and customized support for each interaction step rather than optimizing for a single scenario.

• Experimental or Evaluation Results:

  1. A total of 12 blind and low vision users participated in the study, with results showing:
     • A 30% increase in touchscreen operation success rate.
     • An 11% increase in the success rate of locker search tasks.
  2. NASA TLX evaluations indicated that FetchAid significantly reduced mental and physical workload and frustration during tasks.
  3. Technical evaluations showed that the target detection model achieved an accuracy improvement to 97.7% with data augmentation, and AR navigation precision error was within 10 cm.

• Limitations and Future Directions:

  1. The system's image detection performance under low-light conditions needs improvement, which can be addressed by integrating image enhancement technologies.
  2. Enhancing personalized feedback mechanisms to better accommodate the operational habits of different user groups.
  3. Expanding the application to other public devices (e.g., vending machines, smart diagnostic stations) and other mobile operating systems.
  4. Conducting long-term longitudinal studies to observe users' learning curves and long-term adaptation effects.

This study provides a practical parcel retrieval solution for blind and low vision individuals through innovative tool design, demonstrating the potential of deep learning and augmented reality technologies in accessibility technology. It lays an important foundation for enhancing the inclusivity and societal impact of human-computer interaction technologies.