NaviNote: Enabling In-situ Spatial Annotation Authoring to Support Exploration and Navigation for Blind and Low Vision People

Paper Title

NaviNote: Enabling In-situ Spatial Annotation Authoring to Support Exploration and Navigation for Blind and Low Vision People

Publication Info

• Topic area: Assistive technologies for navigation and spatial understanding for blind and low vision (BLV) users.
• Keywords: Blind and low vision, spatial annotations, navigation, Visual Positioning System (VPS), accessibility, crowdsourcing, voice-based interaction, last-few-meters navigation, multimodal large language models, assistive technology.

Background and Problem

• Problem / challenge: Existing systems for BLV users lack precise localization and effective tools for in-situ spatial annotation authoring. GPS-based systems are limited by low accuracy (10-meter range), and current annotation systems do not empower BLV users to contribute their own knowledge.
• Significance: Accurate navigation and spatial annotation can enhance BLV users' independence, safety, and ability to explore unfamiliar environments.
• Motivation and related work: Prior tools, such as FootNotes and GPS-based navigation apps, have demonstrated the utility of spatial annotations but fail to address the "last-few-meters" navigation problem or support BLV users as active contributors. Emerging technologies like Visual Positioning Systems (VPS) and multimodal large language models (MLLMs) offer opportunities to address these gaps.

Solution

• Proposed approach: NaviNote, a voice-based system combining VPS and MLLMs, enables BLV users to navigate with sub-meter accuracy, query their surroundings, and create in-situ spatial annotations.
• Novelty:
  1. Integration of VPS for sub-meter localization to address the last-few-meters navigation problem.
  2. Voice-based interaction for querying, navigating, and authoring spatial annotations.
  3. A refined taxonomy of spatial annotations tailored to BLV users, including a new "Request" category.
  4. Evaluation of NaviNote's effectiveness in navigation, exploration, and annotation authoring with BLV participants.
• Procedure and key techniques:
  1. Pre-scan environments to generate 3D maps and scene graphs for VPS-based localization.
  2. Use a five-stage interaction pipeline: scanning, localization, querying, navigation, and annotation authoring.
  3. Implement multimodal feedback (audio, haptic, visual) for navigation and annotation access.
  4. Evaluate NaviNote with 18 BLV participants in a public square using navigation tasks and free-form exploration.

Results

• Concrete findings:
  • NaviNote achieved a 14/16 navigation success rate compared to 6/16 for the baseline (TapTapSee).
  • Participants recalled significantly more landmarks with NaviNote (mean difference of 8.75 landmarks, p = 0.001).
  • 82.8% of user queries were correctly answered by NaviNote, with an average response time of 10.8 seconds.
• Advantage over baselines:
  • NaviNote significantly reduced mental demand (p = 0.035), frustration (p = 0.003), and increased perceived performance (p = 0.001) compared to TapTapSee.
  • Hands-free operation and continuous guidance were highlighted as key advantages.
• Experiments / evaluation:
  • Conducted with 18 BLV participants in a public square (~40m x 40m).
  • Tasks included navigation using two systems (NaviNote and TapTapSee), free-form exploration, and annotation authoring.
  • Metrics included navigation success, landmarks recalled, subjective usability ratings (UMUX-LITE, NASA-TLX), and qualitative feedback.
• Limitations and future work:
  • Limited to single-user sessions in a controlled environment; real-world, multi-user, and longitudinal studies are needed.
  • Challenges with annotation alignment over time, crowdsourced scanning, and mid-use localization drift require further investigation.
  • Future work should explore privacy controls, annotation filtering, and mechanisms to ensure annotation reliability.

Summary

NaviNote is a voice-based system that leverages VPS and MLLMs to enable BLV users to navigate with sub-meter accuracy, query their surroundings, and create spatial annotations. Evaluations with 18 BLV participants demonstrated significant improvements in navigation success, usability, and user independence compared to a baseline system. NaviNote also introduced a refined taxonomy of spatial annotations, including a new "Request" category. Future research should address challenges related to annotation alignment, crowdsourced scanning, and real-world deployment in diverse environments.