There is an entire ecosystem of tools, techniques, and processes designed to improve software security by obviating, preventing, preempting, finding, mitigating, and/or eliminating software vulnerabilities. Software vendors have this entire ecosystem to choose from during each phase of the software development lifecycle, which begins when someone identifies a software need, ends when the software vendor decides to halt support for the software, and includes everything in between.
Unfortunately, guidance regarding which of these tools to choose is often non-existent, un-principled, or based solely on anecdotal evidence. In this dissertation, we present three studies to demonstrate that empirical studies can be used to enhance our understanding of the effectiveness of various tools and techniques intended to improve software security.
In our first study, we use a dataset of 9 implementations of the same software specification in order to explore the relationship between web application development tools and the security of the applications developed using those tools. In our second study, we hire 30 code reviewers to perform manual security review of a content management system in an effort to better understand the effectiveness of manual security review as a technique for vulnerability discovery. Finally, in our third study, we analyze a dataset of rewards paid out over the course of two exemplar vulnerability rewards programs (VRPs), in an effort to better understand the costs and benefits of such programs.