Explainable Automatic Evaluation of the Trail Making Test for Dementia Screening

Title of the Paper

Explainable Automatic Evaluation of the Trail Making Test for Dementia Screening

Paper Information

• Subject Area: Automation and explainability in cognitive testing for dementia screening using digital pen technology
• Keywords: Trail Making Test, cognitive assessment, automatic scoring, explainability, multimodal, digital pen

Research Background and Problem Statement

• Problems and Challenges:

  • The Trail Making Test (TMT) is a classic paper-and-pencil cognitive test widely used to assess cognitive function, but traditional scoring methods rely on manual work, which is inefficient and prone to subjective bias.
  • It is difficult to comprehensively record and analyze specific details of patients' performance during the test, such as handwriting features (speed, pressure, pauses, etc.), which may provide additional insights into the patient’s cognitive state.
  • Current digital attempts often rely on fully computerized versions, which differ in clinical validity and consistency from the traditional paper-based TMT and lack system transparency.

• Significance of the Research:

  • Early screening for dementia (e.g., Alzheimer’s disease) is crucial, and TMT provides a simple assessment method.
  • Automated scoring systems can improve efficiency, reduce the burden on healthcare workers, and support subsequent analysis through digital records.
  • Enhancing the system’s explainability helps build trust in the results and improves clinical applicability.

• Motivation and Related Work:

  • Existing studies show that digital pens can capture rich feature data for analyzing handwriting motion characteristics.
  • Some research has attempted to digitize the TMT, but fully computerized methods may require additional clinical validation and face challenges in widespread acceptance.
  • Researchers propose combining traditional paper-and-pencil formats with modern digital methods, using digital pens to record handwriting data and perform automated analysis, thereby preserving the test format’s consistency while achieving comprehensive and transparent analysis.

Solution

• Proposed Method and Framework:

  • This paper proposes a digital pen-based automatic cognitive assessment tool for TMT (including TMT-A and TMT-B).
  • The system conducts tests directly on traditional paper-based TMT forms, capturing real-time handwriting data such as timestamps, trajectories, and pressure through an embedded infrared camera in the digital pen.
  • Automated algorithms generate test scores and structured reports while providing explainable justifications for each scoring decision.

• Innovations:

  • The system does not alter the TMT test format, relying solely on paper-based testing while achieving raw data recording and analysis through digital capture, preserving the test’s clinical validity.
  • It provides real-time scoring and detailed explanations of error types, enhancing transparency and reliability in clinical applications.
  • It measures and analyzes handwriting features (e.g., pause duration, pen pressure, and pen speed), which are challenging to capture manually.

• Implementation Steps and Key Technologies:

  1. System Architecture: The system includes a digital pen, an Android device (tablet or smartphone), a backend data processing service, and integration with the hospital information system (HIS).
  2. Data Collection: The digital pen captures timestamped data, analyzes handwriting trajectories, and identifies the connections between strokes.
  3. Transparent Scoring: The system automatically generates reports, including details on unconnected nodes, incorrect connections, original handwriting rendering, path sequence explanations, and video playback.
  4. Clinical Integration: The system seamlessly integrates with the hospital HIS, allowing test data to be exported in a structured format for doctors to archive and further analyze.

Research Outcomes

• Specific Results:

  • The system accurately measured the primary TMT scoring metric (completion time) and several auxiliary features, such as the number of pen pauses and incorrect connection paths.
  • Automatically generated scores showed high correlation and consistency with manual scoring.
  • Automated analysis revealed significant correlations between handwriting features (e.g., pause duration, pen speed) and TMT completion time, enabling a more comprehensive assessment of patients’ cognitive states.

• Advantages:

  • Unlike fully computerized versions, this system does not alter the traditional TMT test format, using non-intrusive digital methods to retain clinical practicality.
  • Automated scoring reduces the workload for doctors and provides real-time, transparent explanatory reports, making machine decisions easier to understand.
  • The system’s highly modular design supports various types of digital pens and devices, offering strong adaptability.

• Experiments and Evaluation:

  • Experiments were conducted with 40 elderly participants (average age 74.4 years), comparing automated scoring with manual scoring results.
  • Statistical analysis showed high consistency between automated and manual scoring in terms of completion time, number of pen lifts, and incorrect connections.
  • Automated scoring provided additional cognitive feature analyses (e.g., pen speed, pause duration), supplementing standard TMT scoring metrics.

• Limitations and Future Directions:

  • In certain edge cases, the algorithm may be overly strict (e.g., identifying some node connections as errors), requiring further optimization.
  • The system’s handling of interrupted data needs improvement; although test completion time was unaffected, ensuring complete records remains a goal for future work.
  • Future plans include integrating more sensory data (e.g., eye tracking, facial expression analysis, voice processing) to achieve multimodal cognitive assessment.

Conclusion

The proposed digital pen-based automated TMT scoring system achieves real-time scoring, transparent explanations, and data recording without altering the test format, significantly improving the efficiency and reliability of cognitive testing. It also provides potential avenues for future expansion into multimodal cognitive assessment.