EchoWrist: Continuous Hand Pose Tracking and Hand-Object Interaction Recognition Using Low-Power Active Acoustic Sensing On a Wristband

Title of the Paper

EchoWrist: Continuous Hand Pose Tracking and Hand-Object Interaction Recognition Using Low-Power Active Acoustic Sensing On a Wristband

Paper Information

• Research Area: Human-Computer Interaction Technology and Wearable Devices
• Keywords: Acoustic Sensing, Wearable, Smartwatch, Hand Pose, Hand-Object Interaction, Continuous Tracking, Low-Power, Gesture Recognition, Wristband, Machine Learning

Research Background and Problem Statement

• Identified Problems or Challenges:

  1. Human hand activity plays a crucial role in human-computer interaction, yet tracking hand poses and understanding hand-object interactions remain technical challenges.
  2. The high flexibility of the hand and its multi-joint structure increase modeling complexity.
  3. Finger occlusion and interactions with objects can hinder recognition of hand shapes and movements.
  4. Traditional methods rely on external cameras, which are intrusive, power-intensive, and inconvenient to use.
  5. In wearable solutions, skin-contact sensors cause discomfort, camera-based methods have high power consumption and privacy concerns, and many systems lack continuous tracking capabilities or struggle to recognize hand-object interactions.

• Significance: Researching continuous hand tracking and hand-object interaction recognition has broad applications in human-computer interaction, such as smart homes, health monitoring, and augmented reality, enhancing user experience and system functionality.

• Research Motivation and Related Work: The authors explore overcoming limitations of existing technologies by leveraging the low-intrusiveness and multifunctionality of wearable devices. Related fields include gesture recognition technologies based on cameras, electromyography (EMG), and acoustic sensing, though these methods still have room for improvement in terms of power consumption, comfort, and functionality.

Solution

• Proposed Method and Solution:

  1. EchoWrist: A low-power wristband employing active acoustic sensing technology to continuously estimate 3D hand poses and recognize hand-object interactions.
  2. The device is equipped with two speakers emitting silent sound waves and two microphones receiving reflected signals. A deep learning model analyzes echo spectrograms to reconstruct hand poses and recognize daily hand activities.

• Innovations:

  1. EchoWrist is the first wristband to achieve low-power, low-intrusiveness, simultaneous 3D hand pose tracking and hand-object interaction recognition.
  2. It utilizes active acoustic signals to capture hand and object shape and motion information without requiring cameras or complex sensors.

• Implementation Steps and Key Technologies:

  1. Hardware Design: Compact wristband structure with speakers and microphones positioned above and below the wrist.
  2. Acoustic Sensing: Emission of continuous frequency-modulated waves (FMCW) combined with echo analysis to generate echo spectrograms.
  3. Data Processing and Deep Learning: A customized convolutional neural network (CNN) model learns from echo spectrograms to infer hand poses and interactions.
  4. User Studies and Experiments: Collecting user data to evaluate model accuracy and reliability.

Research Outcomes

• Specific Achievements:

  1. EchoWrist can continuously track 20 finger joints with an average Euclidean distance error (MJEDE) of 4.81mm and an average angular error (MJAE) of 3.79°.
  2. Achieved a recognition accuracy of 97.6% across 12 types of daily hand-object interactions.
  3. Power consumption is only 57.9mW, supporting all-day use (19 hours of battery life).

• Comparative Advantages Over Existing Solutions:

  1. Significantly reduced power consumption, only 1/10 to 1/63 of existing technologies, with smaller device size and non-intrusiveness.
  2. The device supports inference for untrained users and unseen gestures, demonstrating high robustness.

• Experimental or Evaluation Results: Experimental results show that compared to existing technologies, EchoWrist significantly improves accuracy, power consumption, and user comfort. Additionally, the device performs stably under varying background noise without degradation in performance.

• Limitations and Future Directions:

  1. Clothing Occlusion: Long-sleeved clothing may affect measurement performance.
  2. Dynamic Motion and Object-Holding Tracking: Future experiments are needed for validation and improvement.
  3. Interaction Recognition Scope: Current experiments focus on kitchen scenarios; future work should expand to broader daily environments.
  4. Privacy Concerns: Further optimization of data filtering is needed to ensure user audio privacy.
  5. Smartwatch Integration: Exploring the application of this technology in existing commercial smartwatch hardware.

Overall, EchoWrist provides a novel approach to gesture recognition in wearable devices, offering significant potential for practical applications. Future research can focus on continuous optimization and functionality expansion to bring the technology to market comprehensively.