HyPockeTuner: Bringing Hyperparameter Optimization to Mobile Devices

Paper Title

HyPockeTuner: Bringing Hyperparameter Optimization to Mobile Devices

Publication Info

• Topic area: Mobile-based Human-in-the-Loop Hyperparameter Optimization (HPO)
• Keywords: Hyperparameter optimization, mobile interface, Human-in-the-Loop, EventCrumb, BOHB, notifications, experiment management, machine learning, visualization, user interaction

Background and Problem

• Problem / challenge: Existing Human-in-the-Loop HPO systems are designed primarily for desktop environments, which limits their usability for long-running experiments. These systems lack effective mobile support, integrated visualizations of experiment history, and mechanisms to reduce the cognitive burden of constant monitoring.
• Significance: Long-running HPO experiments are increasingly common as ML models grow in scale. Effective mobile support could enable users to manage experiments flexibly across different contexts, reducing wasted computational resources and improving user productivity.
• Motivation and related work: Prior systems like ATMSeer and HyperTendril offer desktop-based visualization and control but fail to address mobile usability and the integration of experiment history with user interventions. Commercial tools like W&B and SageMaker provide limited mobile support and lack advanced visualizations for understanding experiment progression.

Solution

• Proposed approach: HyPockeTuner, an interactive mobile system for managing HPO experiments, featuring a notification-driven workflow and a mobile-optimized visualization called EventCrumb.
• Novelty:
  1. Introduction of EventCrumb, a mobile-tailored event sequence visualization that integrates experiment history, user interventions, and outcomes.
  2. Notification-driven workflow to reduce cognitive and physical burdens, enabling timely interventions.
  3. Support for refinement operations (narrowing and redefining hyperparameter spaces) directly from smartphones.
  4. Evaluation through pilot and deployment studies demonstrating usability and real-world applicability.
• Procedure and key techniques:
  1. Design a mobile interface optimized for touch interactions and small screens.
  2. Develop EventCrumb to visualize experiment progress, user interventions, and performance metrics on a unified timeline.
  3. Implement a notification system for real-time alerts on significant events (e.g., performance improvements, system failures).
  4. Extend BOHB to support progressive execution and dynamic narrowing of the search space.
  5. Conduct user studies to validate usability and effectiveness.

Results

• Concrete findings:
  • Pilot study: 96.9% accuracy in task performance using EventCrumb, with a mean task completion time of 27.3 seconds.
  • Deployment studies: Users completed 115 and 163 trials over five days, achieving maximum scores of 64.1 and 0.77, respectively, through iterative refinements and timely interventions.
• Advantage over baselines: HyPockeTuner enabled mobile access, reduced re-entry latency (median = 0–6.2 minutes), and facilitated real-time interventions, which are not supported by existing desktop-based systems or mobile versions of tools like W&B.
• Experiments / evaluation:
  • Pilot study with 12 participants tested EventCrumb’s usability and effectiveness.
  • Deployment studies with two ML experts demonstrated practical benefits in managing long-running HPO experiments.
  • Metrics included task accuracy, interaction counts, re-entry latency, and experiment outcomes.
• Limitations and future work:
  • Limited to specific HPO algorithms (BOHB); generalization to other algorithms requires further work.
  • Current implementation is mobile-only; a hybrid desktop-mobile interface could enhance functionality.
  • Scalability for scenarios with extremely long training cycles (e.g., large language models) remains unexplored.

Summary

HyPockeTuner is a mobile system for managing long-running HPO experiments, featuring EventCrumb, a visualization that integrates experiment history, user interventions, and outcomes. The system reduces cognitive and physical burdens through notification-driven workflows and supports real-time refinements of hyperparameter spaces. Pilot and deployment studies demonstrated its usability and effectiveness, with users achieving significant performance improvements while integrating experiment management into their daily routines. Future work includes extending support to other HPO algorithms, hybrid desktop-mobile interfaces, and applications in educational and collaborative contexts.