ActivitySeeker: Towards Collaborative Personalized Human Activity Discovery and Recognition on Smartphones

Paper Title

ActivitySeeker: Towards Collaborative Personalized Human Activity Discovery and Recognition on Smartphones

Publication Info

• Topic area: Personalized human activity recognition (HAR) using smartphones.
• Keywords: Human activity recognition, smartphone IMU, personalized HAR, collaborative labeling, transfer learning, self-supervised learning, activity discovery, clustering, on-device training, wearable computing.

Background and Problem

• Problem / challenge: Traditional smartphone-based HAR systems face limitations due to single IMU input, diverse phone placements, and high cross-user and within-user variance. Existing approaches either impose high user burden or fail to deliver accurate personalized recognition.
• Significance: Personalized HAR systems are crucial for applications in fitness, health monitoring, and context-aware recommendations, but practical solutions for adaptive, user-specific activity recognition remain underdeveloped.
• Motivation and related work: Prior works either rely on manually labeled datasets, which are burdensome, or use semi-supervised methods with limited performance. Most datasets are collected in controlled environments, failing to capture real-world diversity. There is also a lack of research on recognizing user-specific activity types not seen in training data.

Solution

• Proposed approach: ActivitySeeker, a personalized HAR system that combines self-supervised activity discovery, collaborative labeling, and transfer learning to adapt to individual users on smartphones.
• Novelty:
  1. A collaborative labeling process that reduces user burden by annotating clusters of IMU data with minimal user input.
  2. A lightweight, on-device transfer learning framework for personalized activity recognition.
  3. Integration of time and frequency domain features for robust feature extraction.
  4. A pipeline designed to handle the challenges of wearing diversity and within-user variance.
• Procedure and key techniques:
  1. Data collection and filtering: IMU data is segmented into 5-second windows, with static windows filtered out using a decision tree.
  2. Feature extraction: A ResNet-based model processes both time and frequency domain features.
  3. Clustering: Agglomerative hierarchical clustering groups feature vectors into potential activity clusters.
  4. Collaborative labeling: Users label clusters interactively, significantly reducing manual effort.
  5. Transfer learning: A personalized SVM classifier is trained incrementally on labeled data for activity recognition.

Results

• Concrete findings:
  • ActivitySeeker discovered 95.5% of activity classes and achieved a recognition accuracy of 93.3% in simulated online learning.
  • Collaborative labeling achieved 98.8% accuracy while labeling 67.4% of samples.
  • The system demonstrated an average inference time of 36.3ms and minimal energy consumption (268mAh/hour on a 4200mAh smartphone battery).
• Advantage over baselines:
  • Outperformed non-personalized and continual learning baselines in macro F1-score (0.895 vs. 0.626 and 0.657, respectively).
  • Comparable user experience to the Apple Watch while offering superior activity discovery and recognition capabilities.
• Experiments / evaluation:
  • Simulated online learning with 112.8 hours of real-world IMU data from 13 users.
  • Two user studies comparing ActivitySeeker to manual labeling and the Apple Watch.
  • Ablation studies validating the effectiveness of clustering, transfer learning, and feature extraction components.
• Limitations and future work:
  • Lack of public dataset usage limits reproducibility; authors released their dataset to address this.
  • User interaction process could be further optimized to reduce intrusiveness.
  • Current system focuses on atomic activities; future work could extend to complex activities and static segments.

Summary

ActivitySeeker introduces a novel approach to personalized human activity recognition on smartphones by combining self-supervised learning, collaborative labeling, and transfer learning. The system achieves high recognition accuracy (93.3%) and reduces user burden through efficient labeling and lightweight on-device training. Evaluations demonstrate its superior performance compared to baselines and commercial devices like the Apple Watch, while maintaining a user-friendly experience. ActivitySeeker is well-suited for applications in fitness, health monitoring, and accessibility, with potential for further improvements in long-term usability and complex activity recognition.