Tool-Assisted CVSS Vulnerability Scoring: A Controlled Quantitative Study of Human Assessment

Paper Title

Tool-Assisted CVSS Vulnerability Scoring: A Controlled Quantitative Study of Human Assessment

Publication Info

• Topic area: Cybersecurity vulnerability assessment and decision-support tools.
• Keywords: CVSS, vulnerability scoring, NLP tools, VIET, cybersecurity training, human-centered security, quantitative study, tool adoption, vulnerability assessment.

Background and Problem

• Problem / challenge: Manual CVSS scoring is error-prone, inconsistent, and time-intensive, with expert disagreement and limited scalability. Automated solutions using ML/LLMs face challenges like label inconsistencies and trust issues.
• Significance: Accurate vulnerability scoring is critical for prioritizing remediation efforts, minimizing security risks, and supporting cybersecurity operations.
• Motivation and related work: Previous studies highlight variability in CVSS scoring among experts and the need for tools to improve consistency and efficiency. While NLP-based tools have been explored for extracting security entities, their real-world impact on human analysts remains under-investigated.

Solution

• Proposed approach: VIET, an NLP-based tool that extracts and highlights key entities from vulnerability descriptions to assist analysts in assigning CVSS metrics.
• Novelty:
  1. Application of NLP-based information extraction tools to facilitate manual CVSS scoring.
  2. Development of a user-friendly Web UI for VIET to enable practical use.
  3. Controlled user study with 389 participants to evaluate VIET’s effectiveness across demographics, expertise levels, and vulnerability types.
  4. Analysis of learning effects, showing VIET’s potential as a training tool.
• Procedure and key techniques:
  • Participants evaluated vulnerabilities with and without VIET in a cross-over design.
  • VIET highlighted entities mapped to CVSS metrics (e.g., Vulnerability Type, Privileges).
  • Data collected included accuracy, confidence, time, demographics, and feedback.
  • Regression analyses examined tool effectiveness, expertise, and learning effects.

Results

• Concrete findings:
  • VIET did not improve overall accuracy across all participants but showed benefits for specific metrics (AC, PR, S) and vulnerability types (e.g., CWE-787).
  • Participants using VIET first performed better in subsequent unaided assessments, indicating learning effects.
  • Confidence increased for certain demographics (e.g., male participants, cybersecurity professionals).
• Advantage over baselines: VIET improved accuracy for metrics prone to ambiguity and error, particularly for less experienced participants and specific vulnerability types.
• Experiments / evaluation:
  • 389 participants recruited via MTurk and Prolific.
  • Vulnerabilities included diverse types (e.g., XSS, CWE-787, SQL Injection).
  • Metrics assessed included accuracy, time, confidence, and adoption intent.
• Limitations and future work:
  • Tool effects were small or nonsignificant for experts and certain vulnerabilities.
  • Cognitive load from tool use may offset benefits.
  • Future work should explore interface design, training effects, and applicability to CVSSv4.0.

Summary

This study evaluates VIET, an NLP-based assistive tool for CVSS vulnerability scoring, through a controlled survey of 389 participants. While VIET did not universally improve accuracy, it showed benefits for specific metrics and vulnerability types, particularly for less experienced users. Learning effects suggest its potential as a training tool, enabling users to internalize scoring strategies. Adoption is driven by perceived usefulness, ease of use, and decision support. Findings inform the design of human-centered security tools and highlight areas for future improvement in vulnerability assessment.