"Tell Them They Are a Responsible Entity, Not a Customer": Understanding Practitioner Challenges in Sector CSIRTs

Paper Title

“Tell Them They Are a Responsible Entity, Not a Customer”: Understanding Practitioner Challenges in Sector CSIRTs

Publication Info

• Topic area: Practitioner challenges in operating sector-specific CSIRTs and their service delivery.
• Keywords: Sector CSIRTs, vulnerability notifications, incident response, cybersecurity governance, trust, asset inventories, NIS2 directive, practitioner challenges, service expectations, feedback loops.

Background and Problem

• Problem / challenge: Sector CSIRTs face challenges in aligning their services with constituent expectations, managing dependencies on national CSIRTs, and addressing gaps in feedback mechanisms for critical services like vulnerability notifications.
• Significance: Sector CSIRTs play a crucial role in improving cybersecurity within specific sectors, but operational inefficiencies and misaligned expectations can undermine their effectiveness.
• Motivation and related work: While enterprise CSIRTs and vulnerability notification mechanisms have been studied extensively, sector CSIRTs remain underexplored. Existing guidance focuses on establishing sector CSIRTs but lacks operational insights. This paper addresses these gaps by analyzing practitioner challenges and stakeholder expectations.

Solution

• Proposed approach: A mixed-method study analyzing sector CSIRT services and challenges through interviews, historical data analysis, and a validation workshop.
• Novelty:
  1. First empirical mixed-methods study on sector CSIRTs, focusing on service-specific and strategic challenges.
  2. Detailed evaluation of the vulnerability notification mechanism, revealing systemic issues in notification delivery.
  3. Identification of three key dynamics—resources, legitimacy, and dependency—that shape sector CSIRT operations.
  4. Recommendations for improving sector CSIRT operations, including feedback loops and tailored service strategies.
• Procedure and key techniques:
  • Phase 1: Case study of IBD-CSIRT (Netherlands) with 18 interviews across stakeholders.
  • Phase 2: Historical analysis of vulnerability notifications (2015–2024) and follow-up interviews.
  • Phase 3: Cross-sector validation with 5 additional sector CSIRTs and a workshop with 7 participants.

Results

• Concrete findings:
  • Only 27% of vulnerability notifications from Shadowserver reports reached constituents due to filtering and technical issues.
  • Constituents rarely updated asset inventories, undermining notification effectiveness.
  • Misaligned expectations about incident response led to dissatisfaction among constituents.
  • Services like advisories and expert insights were valued differently depending on constituent maturity and needs.
• Advantage over baselines:
  • Identified systemic gaps in the vulnerability notification pipeline, previously undetected due to lack of feedback loops.
  • Highlighted the dual legitimacy pressures from constituents and governing bodies, which complicate service alignment.
• Experiments / evaluation:
  • Interviews with 26 participants across governance, CSIRTs, and constituents.
  • Analysis of 2,826 tickets and Shadowserver data from 2015–2024.
  • Validation workshop with 7 sector CSIRT practitioners.
• Limitations and future work:
  • Focused on one country and one sector CSIRT, limiting generalizability.
  • Small sample size, though representative of the studied organizations.
  • Future work could replicate the study in other countries, analyze additional services, and assess the impact of improved notification mechanisms.

Summary

This study provides the first systematic analysis of sector CSIRT operations, focusing on service-specific and strategic challenges. Key findings include systemic issues in vulnerability notification delivery, misaligned expectations for incident response, and the impact of resource constraints and dependencies. The study highlights the importance of feedback loops, tailored services, and balancing legitimacy pressures. Recommendations include focusing on the least-capable constituents, extending industry guidelines, and mixing top-down and bottom-up incentives. These insights are crucial for improving the effectiveness of sector CSIRTs in advancing cybersecurity.