Systemization of Knowledge (SoK): Goals, Coverage, and Evaluation in Cybersecurity and Privacy Games

Research Background and Issues

• Identified Problems or Challenges: While gamification approaches in the field of cybersecurity and privacy are widespread, there is a lack of systematic research analyzing their objectives, scope, and evaluation methods. Current studies tend to focus more on educational functions, neglecting a comprehensive summary of overall game objectives and user experience. Additionally, the themes addressed in game design and the systematic nature of evaluations are insufficiently explored.
• Why This Issue is Important: The cost and complexity of cybercrime are rapidly increasing, making it critical to explore innovative ways to enhance cybersecurity awareness and practices, such as through gamification. Effective education and evaluation methods can improve the ability to address these challenges.
• Research Motivation and Related Work: Existing literature includes partial evaluations of educational or gamified designs, but their scope is often narrow, failing to comprehensively cover the diversity of game design objectives and themes. The lack of systematic knowledge summarization regarding objectives, content, and evaluation methods hinders the provision of comprehensive guidance for future research and game design.

Solution

• Proposed Methods or Solutions:
  • The authors conducted a Systematization of Knowledge (SoK) study, systematically analyzing 93 academic studies on gamification approaches in cybersecurity and privacy across three dimensions: objectives, content, and evaluation methods.
  • They introduced three core research questions:
    1. What are the primary objectives of gamification approaches? What cybersecurity and privacy topics do they cover? Who are the target audiences?
    2. What are the evaluation methods and outcomes of these gamification approaches?
    3. How is user experience evaluated, and what are its key components?
• Innovations:
  • The study not only reviewed a broader range of gamification approaches (including educational games, gamification, serious games, etc.) but also systematically integrated their objectives, covered content, and evaluation methods.
  • Evaluation methods were categorized and compared in a layered manner (e.g., interaction-based evaluation, post-game evaluation, and pre-post comparison), refining the perspective on user experience within games.
  • Evidence-based recommendations for game design and evaluation methods were provided, offering practical guidance for future researchers and designers.
• Implementation Steps and Key Techniques:
  1. Literature Collection: Relevant studies were retrieved from five major academic databases (ACM, Scopus, IEEE Xplore, etc.), yielding 3,926 papers.
  2. Literature Screening: Using defined inclusion and exclusion criteria, 93 relevant studies were ultimately identified.
  3. Analysis Process:
     • Open Coding: Data relevant to the research questions were extracted and recorded.
     • Core Connections and Classification: Through brainstorming and team discussions, major themes and their interconnections were summarized.
     • Four categories of objectives (educational behavior, behavior improvement, psychological development, research tools) and eight major content themes (e.g., threats, cryptography, defense strategies) were identified and linked to corresponding evaluation methods.

Research Outcomes

• Specific Achievements:
  • Classification of Game Objectives:
    • Education (e.g., enhancing security awareness, improving educational methods)
    • Behavior Improvement (encouraging behavioral change or improving decision-making abilities)
    • Psychological Development (boosting confidence and security attitudes)
    • Instrumental Research (using games as research tools to analyze specific scenarios)
  • Content Coverage: Spanning 11 core topics, including threat education, cryptography, cybersecurity strategies, and data privacy.
  • Systematic Summary of Evaluation Methods:
    • Three main evaluation methods: interaction-based, post-game testing, and pre-post comparison.
    • Data types: objective data (e.g., scores, task completion) and subjective data (e.g., surveys, user feedback).
    • Mixed methods tailored to different objectives were used to comprehensively evaluate user experience and outcomes.
  • User Experience Elements: Detailed analysis of nine core dimensions (e.g., usability, fun, motivation, cognitive change, etc.).
• Comparison with Existing Approaches and Advantages:
  • Provides a broader and more detailed classification of evaluation methods compared to existing literature, making assessments more systematic and comprehensive.
  • Offers a comprehensive analysis with a larger sample size and diverse methods, surpassing the limitations of traditional literature reviews.
  • Combines user experience evaluation with objective outcome assessments, aiding in the design of more targeted and scalable cybersecurity education games.
• Experimental and Evaluation Results:
  • Games designed with single or multi-objective applications were found to be more effective; for example, games focused on password creation were more impactful in raising security awareness, while complex games covering multiple themes risked causing information overload.
  • While most educational games improved knowledge, they did not significantly enhance behavior, highlighting the need for more direct feedback and behavior-shaping mechanisms.
• Limitations and Future Directions:
  • Current evaluation designs rely heavily on self-reports and experimental settings, potentially underestimating long-term impacts and limitations in real-world applications.
  • Customized games for high-risk groups (e.g., government officials, healthcare professionals) remain an unexplored area.
  • Dynamic game development targeting emerging threats (e.g., AI-driven cyberattacks) could further strengthen the field.
  • Advocates for expanding target audience definitions through analyses of cultural differences or technological literacy backgrounds, particularly involving minority or vulnerable groups.

Through structured analysis, this study not only organizes the objectives and shortcomings of existing game designs but also provides clear guidance on methodologies and future directions. This contributes to establishing more practical and effective pathways for cybersecurity and privacy education.