CMPSC 447: Software Security, Spring 2019

Experiencing Static Analysis

We have discussed that C and C++ applications may suffer from a range of errors from string handling errors to memory errors (e.g., memory leaks and double frees). One effective way of identifying some of these bugs is to run static code analyzers, which statically scan code and output a list of warnings about possible programming bugs and common vulnerabilities. There are many commercial tools such as Fortify, Coverity, and Klocwork that are routinely used by software companies to improve code quality. Those tools are powerful and have GUI interfaces, but unfortunately require license fees. Instead, we are going to experiment with some static code analyzers in the Software Assurance Market (SWAMP); these tools are freely available or allow free uses via academic licenses. Some of these tools may not be as user friendly as the best commercial tools, but they let you get a feel of how this kind of tools works.

For a static code analyzer, two concepts are important: false positives and false negatives. A false positive is a warning given by the analyzer, but is not a real bug. A false negative is a programming bug missed by the analyzer. Ideally, we want a static analyzer with zero false positives and zero false negatives, but this is impossible to achieve in practice. Practical tools have to balance between false positives and false negatives. You will get a feel about the difficulty of this balancing act.

Project tasks

Part 0: Watch this short video about how to use SWAMP.

Sign up for an account in the SWAMP. During the sign up process, you should add a class code to enable your access to commercial tools that are available in the site. Alternatively, if you forgot to use the class code during sign up, you can add the class code to your user profile after the sign up.

I will email you the class code separately. Once the class code is added, SWAMP staff will manually use this class code to authorize you for the use of their commercial tools, during normal business hours, Monday to Friday 9 am to 5 pm CST. So do this step early, as soon as you can.

Part 1 (6 points): Run static analysis tools on your code of homework 3

Add your homework 3 code package to your account and perform an assessment using two tools; specifically you should use the Clang Static Analyzer and the GrammaTech CodeSonar tool. The assessment may take a few minutes to finish. Go over the results by each tool. Note that the results can be viewed in a browser using SWAMP's native viewer or the Code DX viewer. If you use the native viewer, make sure your browser allows pop-up windows to view results. The Code DX viewer gives easier-to-view results, but takes a long time to load in a browser.

Go over the results, and for each one decide whether it is a false positive or a true error in your code. Note that you need to only look at the results on the code you wrote for homework 3, not the template code that was provided to you. Your report should include discussion about the following:

What kind of programming errors did the tools find in your code? It is likely that multiple errors are similar; in this case, you can group the errors and provide an example for each group.
What warnings issued by the tools were false positives in your opinion? Why did you believe they were not programming bugs? Note that CodeSonar provides events (contextual) information about a warning, which may help you assess whether a warning is a false positive or not.
Compare the two tools in SWAMP in terms of what types of errors they each report, their accuracy, performance, and usability.

Part 2 (4 points): Modify your homework 3 per part 1 results

For each true error in your code, modify the code to address the error. In your report, you should explain why your patch would fix the error. It is likely that multiple errors can be fixed in the same way; in this case, you should group the errors in your report, provide an example fix for a group, and say the rest are similar.

Upload your modified homework 3 code to SWAMP and analyze it using the same tools as before. In your report, include a discussion of the new analysis results about whether the old warnings have disappeared after your modification.

What do you need to submit?

A report with the content as described before.
For part 1, provide the warning reports generated by the tools.
Your homework 3 with your fixes applied.

Submit your work through Canvas. It's due on Mar 27th by midnight.