Train as you Fight: Evaluating Authentic Cybersecurity Training in Cyber Ranges

Title of the Paper

Train as you Fight: Evaluating Authentic Cybersecurity Training in Cyber Ranges

Paper Information

• Subject Area: Design of frameworks for cybersecurity training and evaluation
• Keywords: Cyber ranges, evaluation methods, cybersecurity exercises, cyber defense training, learning evaluation framework

Research Background and Problem Statement

• What issues or challenges did the authors identify?

  • Cyber Range Exercises (CRX) provide interactive training in realistic network attack environments, but their evaluation remains a research gap, with existing methods lacking structure and standardization.
  • Most current CRX evaluations focus on specific capabilities (e.g., learning outcomes or technical implementation) while neglecting comprehensive assessments of system value and effectiveness.

• Why is this issue important?

  • As cyberattacks become increasingly complex, developing cybersecurity professionals with practical operational skills is critical.
  • Evaluating exercises to identify strengths and weaknesses is essential for improving cybersecurity training and creating targeted educational content.
  • If the effectiveness of a CRX cannot be validated, institutions or universities are unlikely to implement such security training programs.

• Research Motivation and Related Work

  • This study aims to propose a universal evaluation framework to systematically assess the effectiveness of different cyber ranges and enable developers to improve their exercise designs.
  • A review of existing CRX evaluation methods reveals that current processes often emphasize specific capabilities, such as learning or technical implementation, without a comprehensive standardized framework for overall CRX effectiveness.
  • Related literature discusses learning outcomes and technical features in cyber ranges but lacks exploration of the overall evaluation process and general frameworks.

Solution

• What methods or solutions did the authors propose?

  • The authors proposed a framework called "TARGET" for structured evaluation of cyber range exercises.
  • The TARGET framework includes:
    • A literature-based taxonomy (TARGET Taxonomy) listing 75 evaluation criteria across 8 dimensions: learning impact, learning outcomes, learning experience, team organization, scenario, management, monitoring, and environment.
    • A five-step evaluation process (TARGET Process) guiding implementation, including selecting evaluation criteria, choosing metrics, defining procedures, data collection, and result analysis.

• What is innovative about this solution?

  • It provides comprehensive evaluation standards for cyber ranges, covering all relevant capabilities (learning outcomes, team collaboration, technical implementation, etc.).
  • Through top-down design, the evaluation steps progress from long-term learning goals to detailed technical implementation, ensuring both comprehensiveness and focus.
  • The framework is flexible, allowing customization of criteria and metrics based on the specific needs of different cyber ranges.

• What are the implementation steps and key technologies used?

  1. Literature Review: Screening 59 CRX-related papers from relevant databases to extract evaluation criteria.
  2. Taxonomy Design: Using Kundisch et al.'s taxonomy design method, conducting preliminary categorization, refining classification standards, and supplementing missing criteria to form a complete taxonomy.
  3. Evaluation Process Definition: Clarifying the sequence and methods of the five-step process.
  4. Example Evaluation: Demonstrating the framework's application using the Iceberg CRX developed by the authors.
  5. Data Analysis and Feedback: Evaluating exercise effectiveness through user studies and proposing improvement suggestions.

Research Outcomes

• What specific results were achieved?

  • Developed a systematic CRX evaluation framework (TARGET) capable of assessing the effectiveness of cyber range exercises across multiple dimensions.
  • Introduced a literature-driven taxonomy covering all potential evaluation criteria identified in current research, supplemented with newly defined criteria.
  • Demonstrated the operability of the TARGET framework through its application to the Iceberg CRX, showcasing how it can improve existing cyber range exercises.

• How does it compare to existing solutions?

  • The TARGET framework comprehensively evaluates all key capabilities of CRX, whereas existing methods often focus on single capabilities, such as learning outcomes or technical implementation.
  • It offers a top-down evaluation process, avoiding the disorganized or overly complex issues of traditional evaluation methods.
  • The literature-driven taxonomy ensures both academic rigor and practical applicability.

• What were the experimental or evaluation results?

  • The evaluation of Iceberg CRX showed that the exercise was particularly effective in developing technical skills, such as attack detection and response, with participants' defensive security skills improving by 70%.
  • Participants rated the overall learning experience highly, though some feedback indicated that technical complexity, especially the SIEM tool interface, requires further simplification.

• Limitations and Future Directions

  • Limitations:

    • The literature review only covered certain databases, potentially omitting relevant studies.
    • The current version of the framework provides limited guidance on metric selection and needs further refinement of metric associations.
    • The TARGET process currently supports summative evaluation only and needs expansion to include formative evaluation during the design and development stages.

  • Future Directions:

    • Conduct a broader literature review and investigate commercial cyber range designs to enhance the taxonomy.
    • Provide specific metric lists for each evaluation criterion and map them to the existing taxonomy.
    • Extend the TARGET process to guide cyber range design and innovation stages, enabling developers to dynamically optimize exercises during implementation.