Benchmarking Static C Bug-Checking Tools

One of the problems with the large number of static bug-checking tools is that it is hard for users (developers and managers) to determine which tool best fits their organisation; quantifying precision of a tool and its scalability is necessary. Precision is the ratio of the number of bugs correctly reported to the total number of bugs reported by a tool. Scalability is the ability of a tool to scale proportionally in runtime relative to the size of the input codebase.

Another problem that quality assurance engineers have with these tools is the lack of information on what bugs are missed in the code; quantifying recall of the tool is also needed. Recall is the ratio of the number of bugs correctly reported by a tool to the total number of bugs in a codebase. Taking into account both, precision and recall, gives a measure of a tool's accuracy. Accuracy is the ability of a bug-checking tool to report correct bugs while at the same time holding back incorrect ones.

In Proceedings of "The Second Static Analysis Tool Exposition (SATE) 2009" Workshop, U.S. National Institute of Standards and Technology (NIST) Special Publication (SP) 500-287, June, 2010.