I Can SE Clearly Now: Investigating the Effectiveness of GUI-based Symbolic Execution for Software Vulnerability Discovery

Paper Title

I Can SE Clearly Now: Investigating the Effectiveness of GUI-based Symbolic Execution for Software Vulnerability Discovery

Publication Info

• Topic area: Evaluating the impact of GUI-based interfaces on symbolic execution for vulnerability discovery.
• Keywords: Symbolic execution, GUI, vulnerability discovery, human factors, path explosion, cybersecurity tools, usability, software testing, reverse engineering, API vs GUI.

Background and Problem

• Problem / challenge: Symbolic execution (SE) is effective for software vulnerability discovery but suffers from limited adoption due to high computational demands (e.g., path explosion) and usability challenges. Traditional API-based SE tools require significant expertise, and it is unclear whether GUI-based SE tools improve usability and performance.
• Significance: Addressing these challenges can make SE more accessible and effective for cybersecurity professionals, potentially improving vulnerability discovery outcomes.
• Motivation and related work: Prior research has focused on algorithmic improvements for SE but has largely ignored usability and human factors. GUI-based SE tools have been proposed to reduce cognitive and operational overhead, but their impact on expert workflows and performance remains unexplored.

Solution

• Proposed approach: Conduct a controlled experiment comparing API-based and GUI-based SE tools to evaluate their impact on expert workflows and vulnerability discovery performance.
• Novelty:
  1. First controlled study evaluating GUI-based SE tools for vulnerability discovery.
  2. Identification of differences in workflows and performance between API and GUI users.
  3. Recommendations for improving GUI-based SE tool design based on empirical findings.
  4. Insights into the usability and human factors of SE tools.
• Procedure and key techniques:
  • Recruited 24 vulnerability discovery experts and assigned them to either API-based (angr) or GUI-based (angr-management) SE tools.
  • Designed a realistic vulnerability discovery task based on CVE-2020-25632, ensuring path explosion challenges.
  • Recorded participants’ interactions, analyzed workflows, and compared performance metrics (e.g., task success, time spent, path prioritization effectiveness).
  • Conducted qualitative and quantitative analyses to identify workflow differences and usability challenges.

Results

• Concrete findings:
  • GUI participants were more likely to successfully discover the vulnerability (4/11) compared to API participants (1/13).
  • GUI participants performed more path prioritization operations (10.4 on average) than API participants (4.86 on average).
  • GUI participants completed tasks faster when successful (73 minutes vs. 238 minutes for API participants).
  • GUI-based tools reduced context switching and allowed mid-execution adjustments, enabling more efficient workflows.
• Advantage over baselines:
  • GUI-based SE tools reduced cognitive and operational overhead, enabling more frequent and efficient path prioritization.
  • GUI participants iterated more effectively through configurations and were less likely to give up during the task.
• Experiments / evaluation:
  • Participants were randomly assigned to API or GUI tools and tasked with discovering a heap corruption vulnerability.
  • Data collection included screen recordings, task completion times, and SUS usability scores.
  • GUI tools scored higher on usability (SUS score: 53 vs. 40 for API tools).
• Limitations and future work:
  • Focused on a single vulnerability discovery task; generalizability to other tasks or applications (e.g., software testing) is untested.
  • Study duration was limited, preventing exploration of long-term workflows.
  • GUI design improvements are needed to better support path prioritization and vulnerability triage.
  • Future work could include feature ablation tests, intention analysis, and comparisons with CLI-based SE tools.

Summary

This study evaluates the impact of GUI-based symbolic execution tools on expert workflows and performance in vulnerability discovery. GUI-based tools reduced context switching, enabled mid-execution adjustments, and allowed participants to perform more path prioritization operations, leading to higher success rates and faster task completion compared to API-based tools. However, GUI tools did not improve the effectiveness of individual path prioritizations, highlighting opportunities for further design improvements. These findings underscore the potential of GUI-based SE tools to enhance usability and effectiveness in cybersecurity tasks, providing actionable recommendations for tool developers and researchers.