Google’s Open Source Security Team recently sponsored a fuzzing competition as part of ISCE’s Search-Based and Fuzz Testing (SBFT) Workshop. Our goal was to encourage the development of new fuzzing techniques, which can lead to the discovery of software vulnerabilities and ultimately a safer open source ecosystem. 

The competitors’ fuzzers were judged on code coverage and their ability to discover bugs: 

HasteFuzz took the $11,337 prize for code coverage

PASTIS and AFLrustrust tied for bug discovery and split the $11,337 prize

Competitors were evaluated using FuzzBench, Google’s open source platform for testing and comparing fuzzers. The platform boasts a wide range of real world benchmarks and vulnerabilities, allowing researchers to test their fuzzers in an authentic environment. We hope the results of the SBFT fuzzing competition will lead to more efficient fuzzers and eventually newly discovered vulnerabilities. 

A closer look at our winners

Eight teams submitted fuzzers to the final competition and an additional four industry fuzzers (AFL++, libFuzzer, Honggfuzz, and AFL) were included as controls to represent current practice. 

Code coverage winner – HasteFuzz

HasteFuzz, is a modification of the widely used AFL++ fuzzer. HasteFuzz filters out potentially duplicate inputs to increase efficiency, making it able to cover more code in the 23-hour test window because it is not likely to be retracing its steps. AFL++ is already a strong fuzzer—it had the best code coverage of the industry fuzzers tested in this competition—and HasteFuzz’s filtering took it to the next level.

Bug discovery winners – PASTIS and AFLrustrust

PASTIS makes use of multiple fuzzing engines that can independently cover different program locations, allowing PASTIS to find bugs quickly. AFLrustrust rewrites AFL++ on top of LibAFL, which is a library of features that allows you to customize existing fuzzers. AFLrustrust effectively prunes redundant test cases, improving its bug finding efficiency. Both PASTIS and AFLrustrust found 8 out of 15 possible bugs, with each fuzzer missing only one bug discovered by others. They both outperformed the industry fuzzers, which found 7 or fewer bugs under the same constraints.

Additional competitors, such as AFL+++ and AFLSmart++, also showed improvements over the industry controls, a result we had hoped for with the competition.

Fuzzing research continues

The innovation and improvement shown through the SBFT fuzzing competition is one example of why we have invested in the FuzzBench project. Since its launch in 2020, FuzzBench has significantly contributed to high-quality fuzzing research, conducting over 900 experiments and discussed in more than 100 academic papers. FuzzBench was provided as a resource for the SBFT competition, but it is also available to researchers every day as a service. If you are interested in testing your fuzzers on FuzzBench, please see our guide to adding your fuzzer.

FuzzBench is in active development. We’d welcome feedback from any current or prospective FuzzBench users, your responses to this survey can help us plan the future of FuzzBench.

The Google Open Source Security Team would like to thank the ISCE conference and the SBFT workshop for hosting the fuzzing competition. We also want to thank each participant for their hard work. Together, we continue to push the boundaries of software security and create a safer, more robust open source ecosystem.