DataSentry: Building Missing Data Management System for In-the-Wild Mobile Sensor Data Collection through Multi-Year Iterative Design Approach

Research Background and Problem

• Identified Problems or Challenges: With the widespread application of mobile sensor data in analyzing human behavior (e.g., diagnosing health conditions, predicting productivity), sensor data collected "in-the-wild" often suffers from data loss due to participant behavior or system issues. This data loss leads to a decline in data quality, which in turn affects subsequent modeling and analysis. Existing monitoring systems for detecting missing data mostly focus on individual sensor streams or simple aggregate metrics, without providing comprehensive methods for diagnosing and addressing data loss, especially when sensor data exhibits variability both within and between participants.
• Importance of the Problem: Sensor data loss not only reduces the accuracy of downstream tasks (such as health monitoring or behavior analysis) but also significantly increases the workload for researchers in cleaning and organizing data. Timely detection and handling of data loss are critical for improving research efficiency and ensuring data quality.
• Research Motivation and Related Work: Although several systems have attempted to enhance monitoring capabilities through visualization or modeling, their limitations include:
  1. Difficulty in providing a comprehensive view across participants and sensor types.
  2. Inability to effectively diagnose and address the root causes of data loss. Based on these shortcomings, the authors propose designing an innovative system focused on managing data loss.

Solution

• Proposed Solution: The authors developed a data loss management system called DataSentry, which was refined through three years of iterative design. This system provides researchers with the following features: a comprehensive data view, fine-grained data diagnostic tools, mechanisms for communication with participants, and data loss detection based on within- and between-participant variability analysis.
• Innovations:
  1. Introducing the perspective of "within- and between-participant variability" in the detection and diagnosis of data loss.
  2. Providing user-friendly interactive visualizations to help researchers observe data context and quickly identify the causes of data loss.
  3. Offering various response mechanisms, including automated rule-setting and participant communication support, to facilitate practical operations in sensor data collection.
• Implementation Steps and Key Techniques:
  1. Needs Analysis: Conducting interviews with researchers experienced in mobile data collection to identify system requirements.
  2. Iterative Design: Prototyping based on identified needs and testing the system in real data collection scenarios for continuous optimization.
     • First Round: Rapid prototyping using Tableau.
     • Second Round: Redesigning the system with React.js and D3.js to add more detailed analysis features.
     • Final Round: Introducing new functionalities such as interval comparison views and rule-based participant communication support to improve detectability and communication regarding data loss.
  3. Evaluation and User Feedback: Validating the system's effectiveness and user experience through two rounds of real-world data collection testing and one controlled user study.

Research Outcomes

• Specific Outcomes:
  1. Developed an integrated data loss management system that combines detection, diagnosis, and response mechanisms.
  2. Significantly improved the accuracy of data loss detection by incorporating within- and between-participant variability analysis.
  3. Created a rule-based communication mechanism to reduce the burden of manual participant interaction.
  4. Open-sourced the system's code to support further extension and practical application.
• Advantages Over Existing Solutions:
  1. Provides multi-perspective detection from macro to fine-grained levels, analyzing data variability across different participants and sensors.
  2. Supports cross-validation and comparison of multiple sensor streams during data inspection.
  3. Offers rule-based, automated participant communication mechanisms, enabling users to implement interventions quickly.
• Experimental or Evaluation Results:
  1. Multiple deployment rounds demonstrated the system's effectiveness in identifying and diagnosing data loss issues.
  2. A user study involving 26 participants showed that the full version of DataSentry significantly outperformed traditional methods (e.g., manual data inspection using Python plots) in terms of user experience.
  3. Researchers reported that the system's comprehensive features, from data visualization to participant management, effectively reduced monitoring burdens and improved operational efficiency.
• Limitations and Future Directions:
  1. The system may encounter performance bottlenecks when handling large-scale data and participants, requiring optimization for overall responsiveness.
  2. Enhancing real-time monitoring capabilities and cross-platform compatibility to meet the needs of more practical scenarios.
  3. Expanding the system's rule expression capabilities, such as dynamically setting complex rules (AND/OR conditions) for more precise automation.
  4. While the system has shown promising results in laboratory and small-scale deployment environments, further validation is needed in large-scale, long-term real-world applications.