Smartphone-derived Virtual Keyboard Dynamics Coupled with Accelerometer Data as a Window into Understanding Brain Health

Title of the Paper

Smartphone-derived Virtual Keyboard Dynamics Coupled with Accelerometer Data as a Window into Understanding Brain Health

Paper Information

• Subject Area: Digital Health, Mental Health, Biobehavioral Analysis
• Keywords: Digital Health, Smartphone, Keyboard Dynamics, Accelerometer Data, Cognitive Health, Biomarkers, Mental Disorders, Biobehavioral Analysis

Research Background and Problem

• Problem or Challenge: Traditional clinical assessments rely on self-reports and face-to-face diagnoses, which are subject to recall bias and are not applicable in real-world settings. Although ecological momentary assessment (EMA) improves data timeliness, it still requires active user participation.
• Significance: The widespread use of smartphones enables the passive collection of daily behavioral data (e.g., keyboard dynamics and accelerometer data), providing more ecologically valid and time-sensitive biomarkers for mental and brain health research.
• Research Motivation: The authors aim to investigate human-smartphone behavioral interactions in natural environments using a custom virtual keyboard and accelerometer data to predict cognitive performance and emotional states.

Solution

• Proposed Methods and Solutions:
  • Developed a custom smartphone virtual keyboard to collect keyboard dynamics data (e.g., inter-key delay, typing speed, backspace usage) and synchronized accelerometer data to assess cognitive performance and daily behavioral patterns.
  • Employed sensor fusion methods to combine keyboard dynamics data with accelerometer data for a more comprehensive understanding of users' brain health.
• Innovations:
  • Integrated keyboard dynamics and accelerometer data in natural environments to measure diurnal variations in cognitive performance and analyze time-related behavioral patterns.
  • Provided non-intrusive, high-frequency data sampling for brain health monitoring that is more ecologically valid than traditional clinical assessments.
• Implementation Steps and Techniques:
  1. Data Collection: Used the custom "BiAffect" keyboard to gather user keystroke metadata and accelerometer data while excluding gesture inputs to reduce data variability.
  2. Data Processing: Preprocessed keyboard dynamics and accelerometer data and extracted key variables (e.g., inter-key delay, phone orientation, and motion state).
  3. Statistical Analysis: Applied mixed-effects models to analyze relationships between cognitive performance, time-related behavioral patterns, and smartphone usage dynamics across different time periods.

Research Outcomes

• Specific Findings:
  • Compared to healthy participants, those with mood disorders exhibited lower cognitive performance (assessed using the digital Trail-Making Task Part-B).
  • Keyboard Dynamics:
    • Typing speed (inter-key delay) showed a distinct diurnal pattern among individuals with higher cognitive performance, with faster typing during the day and less pronounced slowing at night.
    • The number of keystrokes correlated with time, with higher-performing individuals demonstrating stronger adherence to circadian rhythms, as their typing activity gradually decreased in the evening.
  • Accelerometer Data:
    • Phone motion intensity decreased over time, showing non-linear diurnal variations.
    • The probability of the phone being in an upright orientation decreased over time, but individuals with higher cognitive performance exhibited a smoother decline.
• Advantages Over Existing Solutions:
  • Provided real-time and ecologically valid data to assess cognitive and emotional changes, avoiding the recall bias inherent in traditional methods.
  • Combined keyboard dynamics and accelerometer data to offer a novel multidimensional perspective on health status assessment.
• Experimental or Evaluation Results:
  • Analyzed over 74,000 typing sessions, with the integrated model evaluating behavioral patterns related to time, cognitive performance, and accelerometer data.
• Limitations and Future Directions:
  • Limitations:
    • The mood disorder group had mild symptoms, limiting the exploration of mood fluctuations' impact on keyboard dynamics and accelerometer data.
    • Using fixed time points instead of individual chronotypes constrained further refinement of diurnal pattern analysis.
  • Future Directions:
    • Incorporate wearable device data (e.g., smartwatches) to explore the relationship between individual chronotypes and behavioral dynamics.
    • Align cognitive assessments more precisely with the time series of keyboard dynamics data to enhance understanding of cognitive changes.

In summary, this study demonstrates the feasibility of using smartphone-collected passive data to assess brain health, paving the way for more precise and personalized behavioral health evaluation methods.