A Usability Evaluation of AFL and libFuzzer with CS Students

Document Title

A Usability Evaluation of AFL and libFuzzer with CS Students

Document Information

• Topic Area: Software security testing, specifically usability studies of fuzz testing tools
• Keywords: Usable security and privacy, fuzz testing, user studies, student participants, tool usability

Research Background and Issues

• Problems and Challenges:

  1. Fuzz testing, as a critical technique for discovering software vulnerabilities, is hindered by its complexity and high demand for specialized knowledge, limiting its use by ordinary developers.
  2. Existing research focuses primarily on performance optimization of fuzz testing tools, with limited attention given to usability issues.
  3. Previous studies (e.g., research by Plöger et al.) have shown that even computer science students and advanced CTF participants face significant difficulties in completing fuzz testing tasks. Most participants failed to complete the tasks, making it challenging to collect meaningful data on subsequent steps in the fuzz testing process.

• Significance:

  • Improving the usability of fuzz testing tools can broaden their application, enabling non-experts to use them more easily. This would reduce the technical gap between professionals and ordinary developers, ultimately enhancing software security.

• Research Motivation and Related Work:

  • The authors aim to address the gap in existing research by conducting a detailed analysis of the usability of two popular fuzz testing tools, AFL and libFuzzer.
  • Through user studies, the research identifies potential usability issues and explores ways to improve the tools.

Solution

Methodology

• Overall Approach:

  • Compare the usability and performance of two mainstream fuzz testing tools, AFL and libFuzzer, through a comprehensive 20-hour user study involving student participants. Analyze each step of the usage process.

• Innovations:

  1. The study design divides tasks into seven distinct subtasks, facilitating data recording and problem diagnosis.
  2. A support system (Hints & Help) is provided to assist participants when they encounter difficulties, expanding the scope of data collection.
  3. Includes a comparative analysis of AFL and libFuzzer, which has not been thoroughly explored in previous studies.

• Experimental Design:

  1. Participants: 47 computer science graduate students with basic C/C++ and Linux skills. Participants were randomly assigned to alternately use the two fuzz testing tools.
  2. Task Allocation: Participants were required to complete seven clearly defined subtasks, including understanding the target program, selecting functional points, writing fuzz target code, and running the testing tools.
  3. Tools and Targets:
     • Testing tools: AFL and libFuzzer.
     • Target programs: Two simple and practical open-source projects (libroxml and tomlc99) were selected.
  4. Scoring and Data Collection:
     • Participants' performance in each step was individually scored.
     • Data collected included directory and command-line history, interview content, and operation logs.

Research Findings

Key Discoveries

1. Success Rates:

   • The success rate for AFL was 29%, while libFuzzer was 14%.
   • Among participants who completed all tasks, only 53% using AFL successfully identified vulnerabilities, compared to 18% for libFuzzer.

2. Usability Issues:

   • Documentation Deficiencies: AFL's documentation was considered difficult to read, while libFuzzer's documentation was slightly better but still challenging for users to understand.
   • Challenging Steps:
     • libFuzzer posed significant challenges in compilation and tool integration (instrumentation), preventing many participants from correctly completing required processes.
     • Memory or address detection tools (e.g., AddressSanitizer, ASAN) created obstacles for both tools.
   • Unfriendly Runtime Information:
     • AFL's runtime output was relatively clear, but crash feedback lacked clarity.
     • libFuzzer's runtime information was less intuitive, though crash feedback was more detailed (e.g., line numbers, function names).
   • Difficulty in Deciding Fuzz Testing Start/Stop: Participants were uncertain whether fuzz testing was running correctly and struggled to determine when testing could be concluded.

3. Participant Performance:

   • Successful participants generally demonstrated higher resilience to frustration, but overall performance was not strongly correlated with programming experience.
   • A small number of participants employed "fuzz target-less" testing of the target program, which performed better with AFL.

Comparative Advantages

• AFL:
  • Slightly weaker documentation support but more intuitive error prompts and clearer runtime feedback.
  • Does not require writing complex fuzz target code, reducing task complexity.
• libFuzzer:
  • More detailed crash output, especially when using ASAN.
  • Overall usability was significantly lower than AFL, with participants encountering more issues during its use.

Experimental Design Innovations

By dividing tasks into subtasks and providing a support system, more participants completed all steps compared to previous studies (e.g., Plöger et al.), offering more detailed data to inform improvements to fuzz testing tools.

Proposed Improvements

1. Enhance Documentation:
   • Include high-level overviews, detailed step-by-step instructions, and comprehensive usage examples.
2. Automate Build Processes:
   • Simplify and automate the compilation and integration of target programs.
3. Improve Error Prompts:
   • Enhance warnings for critical issues such as compilation errors or failed integrations.
4. Strengthen Interface Guidance:
   • Provide visual runtime feedback to simplify result analysis.
5. Refine Fuzz Testing Metadata:
   • Improve metrics related to fuzz testing performance and their interpretability, enabling users to better understand results.
6. Develop Smarter Target-Less Fuzz Testing:
   • Enhance support for workflows that do not require writing fuzz target code.

Limitations and Future Directions

• Limitations:

  • The sample consisted solely of computer science students, which may not fully represent actual developers.
  • Simplified target program design may not accurately reflect real-world development scenarios.
  • Step-by-step task guidance may have mitigated some of the complexities of actual tool usage.

• Future Directions: Extend this methodology to more complex target programs and a broader range of developer groups, including professional software engineers. Additionally, design a new generation of fuzz testing tools based on the recommendations from this study and evaluate their effectiveness with real-world developers.