Summary:

The authors contend that to evaluate security architectures and to be able to compare such architectures we need to be able to speak in quantitative terms regarding them. Current techniques for evaluation do not really allow such quantitative measures. (Red Teams, formal methods, and Qualitative evaluation criteria)

Using probabilistic methods allows the definition of measures that can be directly compared between architectures. It also allows for fairly easily accounting for the likelihood of unknown vulnerabilities, which could effect the measures of interest. Using SAN models (Stochastic Activity Networks) as a basis and a particular Intrusion Tolerant architecture which is currently under development as a case study, they have applied the technique in an attempt to validate certain design choices. The paper then breaks down the architecture into its various components and explains in detail how each is modeled probabilistically. They also assume a model for the attacker and then solve the model using simulation and a model solving tool called Möbius.

Pros:

• Authors make an attempt to quantitatively evaluate a system using their model.
• The approach of using probabilistic models is interesting as it does supply some notion of how the system will respond to unknown attacks.
• Using a solvable model like this to provide some notion of how the system will respond is useful for guiding design requirements of systems. (How good does an IDS have to be before it actually helps with respect to availability?)

Cons:

• Hard to understand the model.
• Authors failed to provide a motivating case for when modeling like this produces non-intuitive results.
• Attacker behavior is not supported by real world data.
• Is behavior model able using exponential distributions? There was some concern that the attacker model was not accurate and further could not be accurate even with extension, but was inherently wrong because it was based on exponential distributions.
• Overall hard to reason about the model’s effectiveness when the system evaluated has not been evaluated by any other means.

Votes:

• Strong Accept: 0
• Accept: 9
• Reject: 5
• Strong reject: 0