OpenCD: Empowering Diagnosis of Children's Mathematical Cognition through Open-ended Multimodal Tasks

Paper Title

OpenCD: Empowering Diagnosis of Children's Mathematical Cognition through Open-ended Multimodal Tasks

Publication Info

• Topic area: Cognitive diagnosis in early mathematics education using AI-assisted multimodal analysis.
• Keywords: Cognitive diagnosis, multimodal tasks, early mathematics, open-ended assessment, Evidence-Centered Design, Vision-Language Models, teacher-AI collaboration, explainable AI, formative assessment, process-based analysis.

Background and Problem

• Problem / challenge: Existing assessment methods in early mathematics education often rely on closed-ended questions, which fail to capture nuanced cognitive processes. Open-ended tasks provide richer insights but are challenging for teachers to analyze systematically due to their unstructured and diverse nature.
• Significance: Understanding children's mathematical cognition is critical for targeted teaching interventions, as foundational deficits in early education can impact long-term learning outcomes.
• Motivation and related work: Prior research highlights the importance of multimodal responses and embodied cognition in mathematics learning. However, current AI systems focus on correctness in structured tasks, neglecting the nuanced cognitive behaviors in open-ended assessments. This paper addresses the gap by designing a system to automate and explain the analysis of multimodal student responses.

Solution

• Proposed approach: OpenCD, an AI system designed to analyze multimodal open-ended tasks and provide teachers with transparent cognitive diagnoses and actionable insights.
• Novelty:
  1. Integration of Evidence-Centered Design (ECD) to ground AI diagnostics in pedagogical theory.
  2. Hybrid architecture combining Vision-Language Models (VLMs) and rule-based expert models for interpreting diverse student responses.
  3. Transparent teacher-facing interface with bi-directional traceability linking diagnostic conclusions to behavioral evidence.
  4. Class-level analytics for collective instruction alongside individual profiles for personalized intervention.
• Procedure and key techniques:
  • Multimodal data (e.g., drawings, object manipulations, speech) is captured and converted into text-and-image scripts for analysis.
  • VLMs interpret response processes and detect behaviors, while rule-based modules map these behaviors to cognitive nodes.
  • Evidence propagates through a cognitive graph to synthesize holistic diagnoses.
  • Diagnoses are visualized in cognitive graphs and comprehensive reports, enabling teachers to trace conclusions back to specific student actions.

Results

• Concrete findings:
  • Diagnostic accuracy: 90.3% of AI-generated diagnoses rated as "completely reasonable" by expert teachers.
  • Error analysis: Majority of errors stemmed from conservative biases, such as underestimating mastery or misjudging evidence sufficiency.
  • User study: Teachers reported reduced mental demand (p=.046) and effort (p=.0045) when using OpenCD, with increased confidence in their diagnoses (p=.0081).
• Advantage over baselines:
  • Automates analysis of unstructured multimodal data, addressing scalability challenges in process-based assessment.
  • Provides deeper insights into student cognition compared to manual analysis.
• Experiments / evaluation:
  • Expert evaluation: Two experienced teachers reviewed diagnoses for 320 cognitive nodes across 16 students.
  • User study: 20 teachers analyzed student responses under manual and OpenCD-assisted conditions, with qualitative and quantitative feedback collected.
• Limitations and future work:
  • Limited generalizability to higher-grade topics and less digitized contexts.
  • Latency in the diagnostic pipeline (~1 minute per response) precludes real-time adaptive questioning.
  • Need for scalable methods to construct cognitive graphs and refine intermediate analysis steps.

Summary

OpenCD addresses the challenge of analyzing children's multimodal open-ended responses in early mathematics education by combining Vision-Language Models with rule-based expert systems grounded in Evidence-Centered Design. The system achieves high diagnostic accuracy (90.3%) and significantly reduces teachers' cognitive burden while enhancing their insights into student thinking. Through transparent interfaces and class-level analytics, OpenCD empowers teachers to make informed pedagogical decisions and supports scalable process-based assessment. Future work aims to expand content coverage, optimize latency, and validate long-term impacts on teaching and learning outcomes.