A novel interaction for competence assessment using micro-behaviors: Extending CACHET to graphs and charts

Title of the Paper

A novel interaction for competence assessment using micro-behaviors: Extending CACHET to graphs and charts

Paper Information

• Subject Area: Human-Computer Interaction (HCI), Learning Analytics, Schematic Understanding of Images and Charts
• Keywords: Competence Assessment, Chunking, Task Analysis, GOMS Model, Individual Differences, Learning Analytics, Chart Understanding, Data Visualization

Research Background and Problem

• Identified Problems or Challenges:

  • Current methods for assessing learning competence are mostly based on traditional linear symbols (e.g., mathematical formulas or code), while the evaluation of two-dimensional charts and nonlinear symbols remains an underexplored area.
  • Individuals exhibit significant differences in methods and strategies when transcribing data from two-dimensional charts, which severely impacts assessment accuracy.
  • Educational data mining often fails to effectively integrate individual cognitive characteristics, such as the manifestation of "chunking" during learning processes.

• Importance of the Problem:

  • Charts and data visualizations are widely used tools in educational and professional fields, and the ability to understand and utilize these representations directly affects learning and work performance.
  • Better assessment of individual competence in these areas is crucial for designing educational curricula and personalized learning plans.

• Research Motivation and Related Work:

  • The CACHET method proposed by Cheng (2014, 2015) analyzes chunk structures in memory by identifying micro-behaviors during competence assessment.
  • Previous studies have demonstrated the significance of chunking theory in chess memory, geometric figure drawing, and mathematical formula transcription, but it has not been well applied to chart comprehension and complex two-dimensional data presentations.
  • Existing methods based on machine learning and handwriting analysis have explored related areas, but they typically require large datasets and fail to fully utilize chunking theory.

Solution

• Proposed Method or Solution:

  • A novel interactive method, TIPS (Transcription with Incremental Presentation of the Stimulus), is proposed. It incrementally presents stimulus materials, dynamically displaying and restricting transcription task strategies to reduce variability.
  • Six charts with varying complexity and familiarity levels were designed and tested, and participants' micro-behavior data during the "viewing-drawing" phases were evaluated.

• Innovative Contributions:

  • The CACHET method was extended for the first time to assess competence in understanding charts and other two-dimensional graphics.
  • TIPS reduces strategy noise by incrementally displaying stimuli, generating more distinct chunking signals.
  • The CPM-GOMS model was introduced into task analysis to interpret participants' micro-behaviors and predict competence-related pauses (e.g., viewing pauses, drawing pauses).

• Implementation Steps and Key Techniques:

  1. Task Design and Data Collection: Stimuli included a simple house diagram, a traditional Rey figure, and simple/complex line and bar charts, presented in stages via TIPS.
  2. Measurement and Recording: Continuous micro-behaviors were recorded, including:
     • Number and duration of viewings;
     • Drawing time;
     • Interruptions and pause delays (e.g., View Pause and Draw Pause).
  3. Modeling and Analysis: The CPM-GOMS model was proposed to explain cognitive operations during task completion, combined with statistical analysis to explore chunking behaviors and individual differences.

Research Findings

• Specific Results:

  • Validation of Chunking Signals: Familiar stimuli formed larger chunks (higher viewing episode size) and were associated with longer viewing and drawing times.
  • Behavioral Characteristics Assessment:
    • View Pause results indicated that the cognitive familiarity of stimuli significantly influenced automatic perception and comprehension processes.
    • Draw Pause better reflected cognitive load related to information integration and chunk generation.
  • Model Validation: The CPM-GOMS model successfully predicted and explained pause distributions in participants' micro-behaviors across different stimuli, with a prediction error (MAPE) below 10%, demonstrating high reliability.

• Comparison with Existing Solutions:

  • Compared to the traditional CACHET method, TIPS effectively addresses noise issues caused by strategy differences in transcribing two-dimensional data.
  • TIPS collects data at a finer granularity than machine learning methods and does not require large-scale training datasets.

• Experimental or Evaluation Results:

  • In the experiment, six computer science participants exhibited significant chunking behaviors during transcription tasks, particularly with familiar graphics (house diagram and simple charts), where chunking signals were more pronounced than with unfamiliar Rey figures.
  • ANOVA analysis revealed that stimulus complexity, individual differences, and task type significantly influenced pause durations (p<0.05).

• Limitations and Future Directions:

  • Sample Size Limitation: The study only recruited six participants, resulting in a small sample size and limited generalizability of the conclusions.
  • Task Complexity Challenges: For complex stimuli (e.g., charts with extensive details), the interactions in TIPS require numerous operations, increasing task completion time and cognitive load on participants.
  • Future Directions:
    • Expand the experimental sample to include individuals with diverse cognitive abilities and backgrounds.
    • Optimize TIPS user interactions to reduce task complexity.
    • Explore the practical application of TIPS in educational settings, such as assessing students' understanding of data visualizations.