CT4ALL: Towards Putting Teachers in the Loop to Advance Automated Computational Thinking Metric Assessments in Game-Based Learning

Research Background and Issues

• What issues or challenges did the authors identify?
  The authors pointed out that assessing students' computational thinking (CT) skills in open-ended learning environments based on constructivism and game-based learning (GBL) is challenging both technically and conceptually. Specifically, existing automated CT assessment tools (e.g., Dr. Scratch) face the following issues:

  1. Limitations in universal assessment: There is a conflict between standardized, one-size-fits-all assessment methods and students' personalized design practices.
  2. Lack of teacher involvement: Existing research often focuses solely on optimizing the backend of tools, neglecting the user experience and needs of teachers.
  3. Technical limitations: Current assessments struggle to capture the nuanced and complex dimensions of CT skills and fail to support multi-project comparisons and CT development tracking.
  4. Insufficient definition and standardization: The ongoing development of CT theory limits the applicability of assessment tools across different learning contexts.

• Why is this issue important?
  CT is considered a core skill for 21st-century learning and digital literacy. Particularly in STEM and computer science education, CT plays a decisive role in cultivating students' algorithmic thinking and problem-solving abilities. However, effective automated tools for assessing CT skills acquired through game-based and open-ended exploratory learning paths are still lacking. Enhancing the transparency, flexibility, and adaptability of these assessment methods can significantly improve CT learning outcomes in K-12 education.

• Research Motivation and Related Work
  The authors were motivated by the limited perspectives of existing work on automated CT assessment (e.g., focusing solely on technical implementation while neglecting multidimensional CT development) and recognized the necessity of treating teachers as key stakeholders. The research aims to expand the functionality and assessment capabilities of Dr. Scratch, enabling it to:

  1. Better adapt to students' learning differences;
  2. Capture new dimensions of CT skill improvement;
  3. Enhance tool usability and interpretability through teacher involvement.

Solution

• What methods or solutions did the authors propose?
  The authors developed a new assessment tool—Dr. Scratch 2.0—which recalibrates and extends the original Dr. Scratch by adding two new CT dimensions ("Mathematics and Strings" and "Motion") and five new functional modes, including:

  1. Personalized Rubric: Allows teachers to adjust CT assessment criteria to accommodate different design preferences.
  2. Comparison Mode: Supports comparative analysis of different students' projects.
  3. Batch Assessment Mode: Enables rapid analysis of an entire class's projects.
  4. CT Tracking: Records and visualizes the temporal evolution of CT skills within projects.
  5. Recommender Assistant: Provides intelligent guidance for students by offering feedback on code design quality issues (e.g., code redundancy).

• What are the innovative aspects of this solution?

  1. Introduction of new dimensions: By adding "Mathematics and Strings" and "Motion," the tool addresses gaps in traditional CT assessments.
  2. Support for personalized learning: The personalized rubric offers flexibility, allowing assessment criteria to be tailored to students' individual design needs.
  3. Support for longitudinal learning records: The CT tracking feature provides teachers and students with concrete pathways for assessing and improving CT skills.
  4. Human factor integration: The design emphasizes the teacher's perspective and incorporates their feedback into the core design process.

• What are the implementation steps and key technologies used?

  1. Expansion of the existing scoring mechanism: Extending the original 3-point scale to a 4-point scale to capture higher-level CT skills.
  2. Addition of new functional interfaces: Designing user-friendly front-end interfaces for each extended mode, supporting project uploads, assessments, and feedback display.
  3. Qualitative and quantitative validation: Conducting teacher interviews (n=7) to gather feedback and iteratively optimize the tool's usability and functional design.
  4. Combining computational and semantic analysis methods: Enhancing the recommender system's intelligence by employing techniques that capture the semantic aspects of students' code designs (e.g., "refactoring suggestions").

Research Outcomes

• What specific outcomes were achieved?

  1. Successfully developed and implemented the extended version of Dr. Scratch 2.0, demonstrating through comparative studies with the original Dr. Scratch that the new tool can more comprehensively capture students' CT skill development.
  2. Collected multidimensional feedback from teachers on the tool, particularly regarding current educational needs and directions for improvement.
  3. Demonstrated the potential of innovative features (e.g., personalized rubric, CT tracking, and recommender assistant) to enhance CT assessment and teaching.

• What advantages does it have compared to existing solutions?

  1. Officially integrates teachers' and students' personalized needs into the CT assessment process.
  2. Provides more flexible support for complex coding practices (e.g., dynamic event synchronization and advanced mathematical operations).
  3. Decentralized assessment: Not only emphasizes the final product but also promotes consideration of students' design processes.
  4. Interface optimization and intuitive result display significantly improve the tool's usability.

• What were the experimental or evaluation results?

  1. Teachers generally indicated that the tool helps reduce the burden of assessment, especially in addressing the challenges posed by students' varying cognitive levels.
  2. Results showed that the tool's new features and CT dimensions can more finely reflect students' CT skill levels in game design.
  3. However, teachers noted that for students and non-computer science teachers, the learning curve for certain CT terminology and related features remains too steep.

• Limitations and Future Directions

  1. Small sample limitation: Only seven teachers were interviewed, and the results may not represent all educational contexts.
  2. Balancing automation and interpretability in technical expression: Teachers suggested further improvements in terminology expression and the development of video resources suitable for younger students.
  3. Functionality expansion: Future research should provide deeper support for Scratch Remix and collaborative programming, as well as quantitative validation of tool performance (e.g., comparisons with other CT tools).
  4. Inclusivity: Exploration is needed to adapt the tool for rural schools or underrepresented students in CS education, further advancing the goal of CT4All.

In summary, this study demonstrates the necessity of treating teachers as key stakeholders in improving CT assessment and takes a significant step forward with the implementation of Dr. Scratch 2.0, providing a more flexible and transparent assessment tool for K-12 STEM and CS education.