Summary: This paper takes a case study approach to analyzing software security. The
authors look at the OpenBSD kernel over the past 7.5 years and 15 versions and
examine reported vulnerabilities and fixes according to data extracted from the
OpenBSD publicly accessible CVS repository and public vulnerability databases.
They attempt to answer the following questions:

Q1. How much does legacy code influence security today? A: A lot. 62% of
vulnerabilities were from foundational code.
Q2. Do larger code changes have more vulnerabilities? A: Didn't find a
significant correlation.
Q3. Do today's coders introduce fewer vulnerabilities per line of code? A: While
there is some indication that code providing security functionality is more
vulnerable, this is likely because its more stringently attacked.
Q4. What is the median lifetime of a vulnerability? They calculated the median
lifetime of a reported vulnerability, and found it was 2.6 years for
foundational vulnerabilities. Surprisingly high!
Q5. Are vulnerability reporting rates declining? Using the Laplace test and
assuming failures come according to a Poisson process, their results show that
foundational vulnerabilities are being reported at a declining rate.

The reason I selected this paper is that I think it is very offbeat to what we
are normally used to in SRG I thought it would lead to interesting discussion.

Pros:
This was a very interesting topic and analysis; They were very extensive in
their method and the work does an impressive attempt at analyzing the evolution
of software. The authors did a lot of painstaking work to get their dataset and
did an excellent job stating their assumptions. We liked their predicition for
how many bugs are left in the OpenBSD kernel.

Cons:
The title is misleading, though cute. It should have been "Analyzing the
software security of the OpenBSD kernel." We would have liked further discussion
on why its important to analyze OpenBSD and we can compare these results with
other OSes. It may have been better to choose a different kernel that is more
widely used, and thus scrutinized. Some didn't like the analysis of bundled
packages, as it was weaker than the analysis of the kernel as a whole. The
authors could have used a different tool than diff to improve their false
positive rate with respect to code changes. They could have measured severity
categories of vulnerabilities by defining a small set of categories and tagging
them.

Questions:
The findings in this paper seem to imply that estimates can be used to gauge
some amount of further testing required to meet reliability requirements. Do you
think this is possible?
A: Maybe. It would be an interesting statistic that companies might be
interested in. For example, this might be a worthwhile investment for Microsoft.

So, it took six years to find half of the known vulnearbilities on a system that
is strictly focused on security. Does this give us a dismal view of software
security?
A: Yes, especially considering the strict security practices of OpenBSD
developers. It would be interesting to compare this analysis with other kernels
that don't take as focused of an approach to security and see if the results are
worse.

What is your opinion on the social utility of public vulnerability disclosure?
A: Public disclosure is a good thing because it forces companies to take
responsibility for flaws in their products, whereas they might ignore other
means of disclosure. Responsible disclosure is another method that is perhaps
more considerate to companies.

Is there anything you would have liked to see from this study? It would have
been most interesting to see if specific authors were responsible for more or
less numbers of vulnerabilities.
A: We would have liked to see a better outline of the contributions they were
trying to make. The discussion in our reading group helped, but this should have
been clear from the paper. Also, they should have included the severity and
maybe types of vulnerabilities. This could have given us a higher granularity on
what types of bugs cause vulnerabilities and how long it takes certain types to
be found.

One of the methods they mention in related work is measuring software security
through market-mechanisms. To some extent, markets exist already. People sell
vulnearbilities on ebay, or there are more structured services like VeriSign
iDefense and 3COM TippingPoint where people sell vulnearbilities and these
services sell them to companies. What are people's thoughts on this?
A: We discussed how market pressures can improve software security and how
companies can leverage this to estimate how much they should invest in software
testing.

VOTE:
Weak Accept: 10
Strong Accept: 5