The Effects of Group Discussion and Role-playing Training on Self-efficacy, Support-seeking, and Reporting Phishing Emails: Evidence from a Mixed-design Experiment

Title of the Paper

The Effects of Group Discussion and Role-playing Training on Self-efficacy, Support-seeking, and Reporting Phishing Emails: Evidence from a Mixed-design Experiment

Paper Information

• Research Area: Cybersecurity, Information Security Education
• Keywords: Self-efficacy, Support-seeking, Reporting Phishing Emails, Role-playing Training, Group Discussion, Mixed-design Experiment, Anti-phishing Training

Research Background and Problem

• Identified Issues or Challenges:

  • Phishing attacks are a common form of cybercrime that exploit human vulnerabilities through social engineering to obtain confidential information or infect devices.
  • Many employees within organizations lack sufficient vigilance in dealing with phishing emails, especially when technical defenses fail, leaving human defenses as the last line of protection.
  • Existing phishing email education programs show inconsistent results, with embedded simulated phishing tests often failing to significantly improve employees' cybersecurity defenses.

• Importance of the Problem:

  • Phishing attacks often serve as entry points for more severe cyberattacks, such as ransomware or intellectual property theft, causing significant economic losses for organizations annually.
  • Enhancing employees' awareness and ability to defend against phishing emails is urgently needed and represents a critical step in improving organizational information security.

• Research Motivation and Related Work:

  • Existing studies suggest adopting more interactive and practice-oriented anti-phishing training methods.
  • Previous research on phishing training involving role-playing and group discussions has shown promising potential but lacks validation in real workplace scenarios.
  • This study designs and validates these two interactive training methods, focusing on their impact on improving employees' self-efficacy, support-seeking intentions, and phishing email reporting behaviors.

Solution

• Methods and Solution:

  • Designed and implemented two training methods: group discussion and role-playing.
  • Group discussion: Employees share phishing experiences and discuss how to identify phishing emails and effective response strategies.
  • Role-playing: Participants simulate hackers designing phishing emails, while other groups identify whether the emails are suspicious.

• Innovative Contributions:

  • Introduced support-seeking intention as a new metric for evaluating anti-phishing training effectiveness.
  • Scientifically assessed the training effects through a mixed-design experiment (including three control groups, three measurement time points, and simulated phishing tests).

• Implementation Steps and Techniques:

  1. Designed two training programs based on real phishing email cases with interactive two-way communication.
  2. Recruited and randomly assigned 105 participants to the group discussion, role-playing, and control groups.
  3. Conducted surveys with participants before training (Q1), immediately after training (Q2), and seven days later (Q3).
  4. Designed and sent three rounds of simulated phishing tests during the experiment to evaluate participants' behaviors and responses in real work environments.

Research Outcomes

• Specific Findings:

  • Both group discussion and role-playing significantly improved participants' self-efficacy and support-seeking intentions.
  • Role-playing training had a stronger effect on enhancing support-seeking intentions compared to group discussion.
  • Both training methods led participants to report more phishing emails during simulated phishing tests, with significantly higher reporting rates compared to the control group.

• Comparison with Existing Solutions and Advantages:

  • Compared to simple online education or embedded phishing tests, these two training methods are more interactive and better suited to real workplace scenarios.
  • Role-playing, through the "simulated hacker" approach, encouraged participants to understand phishing emails from the attacker’s perspective, enhancing their awareness and willingness to proactively seek support.

• Experimental or Evaluation Results:

  • Group discussion immediately improved self-efficacy (p < 0.001), and this improvement remained significant after seven days.
  • Role-playing significantly enhanced support-seeking intentions after seven days (p < 0.01) and moderately reduced participants' behavior of "clicking phishing links."
  • Over 80% of participants indicated that both training methods were practically helpful for their work and expressed willingness to recommend them to colleagues.

• Limitations and Future Directions:

  • Limitations include the study being conducted in a university environment, which may not be universally applicable; participants tended to have stronger technical backgrounds.
  • Long-term effectiveness and potential side effects of overconfidence were not thoroughly explored.
  • Future work may include:
    1. Long-term tracking of training effects and validation in other organizational settings.
    2. Exploring better integration of storytelling and security skill enhancement.
    3. Further optimization to reduce the "ceiling effect" for high-performing participants and extend applicability to broader demographic backgrounds.