VueBuds: Visual Intelligence with Wireless Earbuds

Paper Title

VueBuds: Visual Intelligence with Wireless Earbuds

Publication Info

• Topic area: Egocentric visual intelligence integrated into wearable devices.
• Keywords: Wireless earbuds, egocentric vision, vision language models, wearable AI, binocular cameras, real-time processing, low-power systems, user studies, privacy, accessibility.

Background and Problem

• Problem / challenge: Wireless earbuds lack visual capabilities, limiting their potential for multimodal interaction compared to smart glasses and other wearable devices. Challenges include integrating cameras within strict size, weight, and power (SWaP) constraints and addressing facial occlusion from ear-level cameras.
• Significance: Wireless earbuds are one of the most ubiquitous wearable platforms, with a user base 150–200x larger than smart glasses. Adding visual intelligence to earbuds could democratize access to AI-powered visual assistance.
• Motivation and related work: Prior wearable systems like smart glasses and AR headsets have explored visual intelligence but face adoption barriers due to social discomfort and limited user bases. Ear-worn devices have evolved with sensors for health and motion tracking but lack egocentric vision capabilities. Existing camera systems struggle with power consumption and privacy concerns, leaving a gap for compact, low-power camera-enabled earbuds.

Solution

• Proposed approach: VueBuds, a camera-integrated wireless earbud system that streams visual data to a host device for real-time processing by vision language models (VLMs).
• Novelty:
  1. Development of the first dual-camera earbud prototype operating under 5mW power with minimal battery overhead.
  2. Binocular vision system to mitigate facial occlusion and provide comprehensive forward coverage.
  3. End-to-end system optimizations for real-time multimodal interaction with VLMs, achieving under 3-second latency.
  4. Comparative evaluation showing performance parity with smart glasses across visual question answering tasks.
• Procedure and key techniques:
  • Integration of low-power Himax HM01B0 CMOS cameras into Sony WF-1000XM3 earbuds using custom PCBs and 3D-printed enclosures.
  • Binocular camera orientation and windowed readout to reduce facial occlusion and expand field of view.
  • BLE-based wireless streaming pipeline with opportunistic image stitching to reduce latency.
  • On-device processing using Qwen2.5-VL for scene understanding, OCR, and translation tasks.

Results

• Concrete findings:
  • VueBuds achieved 82.5% accuracy in object recognition, 94.3% in OCR, and 83.8% in translation tasks during in-person studies.
  • End-to-end latency reduced to 1.14 seconds with opportunistic stitching.
  • Battery life impact limited to 11–14% under high usage (60 queries/hour).
• Advantage over baselines:
  • Comparable response quality to Ray-Ban Meta smart glasses (Mean Opinion Score: VueBuds 3.33 vs. Ray-Ban Meta 3.32).
  • Broader accessibility, with 93.3% of participants using earbuds versus 62.7% using glasses.
• Experiments / evaluation:
  • Online study (n = 74) comparing response quality across 17 VQA tasks.
  • In-person study (n = 16) assessing real-world performance in diverse environments.
  • System benchmarks for power consumption, latency, and field of view.
• Limitations and future work:
  • Monochrome imaging limits OCR performance on fine text and object recognition under adverse lighting.
  • Lack of gaze tracking and multi-turn conversational capabilities.
  • Privacy concerns related to bystander awareness and inferential disclosure.
  • Future directions include color imaging, adaptive camera positioning, gesture-based interaction, and privacy-preserving techniques.

Summary

VueBuds introduces camera-integrated wireless earbuds as a novel platform for egocentric visual intelligence, leveraging binocular vision and vision language models for real-time interaction. The system achieves comparable performance to smart glasses while addressing accessibility barriers, with minimal impact on battery life and latency. User studies validate its feasibility for tasks like scene understanding, OCR, and translation, though challenges remain in resolution, privacy, and interaction modalities. VueBuds positions earbuds as a promising wearable form factor for visual AI applications.