In Search of
Evidence-Based IT-Security
Hanno Böck
https://hboeck.de
Introduction
Hanno Böck, freelance journalist and hacker.
Writing for Golem.de and others.
Fuzzing Project, funded by Linux Foundation's Core Infrastructure Initiative.
Author of monthly Bulletproof TLS Newsletter.
IT Security Products
Picture: Hanno Böck, Black Hat USA 2016
Picture: Hanno Böck, Black Hat USA 2016
Picture: Hanno Böck, Black Hat USA 2016
Picture: Hanno Böck, Black Hat USA 2016
I'm a bit skeptical
Sources: PCWorld, The Register, April King/Twitter, Justin Schuh/Twitter.
New research: Comparing how security experts and non-experts stay safe online (Google Security)
There is considerable disagreement whether many IT security and especially Antivirus products are a good idea.
How do we know if these things work?
Let's look at something completely different
Another Anti-Virus
Pictures: CDC/Wikimedia Commons, Hanno Böck
Vitamin C against the Common Cold
It's probably not very useful.
Why do we know that?
SCIENCE!
Vitamin C
Regular ingestion of vitamin C had no effect on common cold incidence in the ordinary population [...]. However, regular supplementation had a modest but consistent effect in reducing the duration of common cold symptoms [...].
Trials of high doses of vitamin C administered therapeutically, starting after the onset of symptoms, showed no consistent effect on the duration or severity of common cold symptoms.
Randomized Controlled Trial (RCT)
Randomly split patients in groups.
Simple: Group A gets medication, Group B gets placebo
More complex: Group A gets new medication, Group B gets best old medication, Group C does alternative to medication, e.g. exercise.
See who gets better.
Meta analyis
We don't care about single studies. We care about all the evidence we have.
Meta analysis: Pool results from all available studies.
Evidence-based Medicine
Ideally all decisions in medicine should be based on high quality scientific evidence.
Science has problems
Sources: PLOS One, Nature News, STAT
Spotting bad science (1)
Small number of research subject (underpowered studies).
Making causal claims although the data only supports a correlation (too many results purely rely on observational data).
Single or few studies. Good science needs to be replicated.
Spotting bad science (2)
Publication bias - only studies with "postive" results get published.
Fishing for results and outcome switching. (If we don't find X in our data, maybe we find something else.)
Ideally all empirical studies should be preregistered (but we're very far from that).
Now let's get back to IT security
This was an empty slide.
It was also the complete list of all randomized controlled trials ever done on the effectiveness of Antivirus applications or other IT security products.
Questionable Antivirus tests
There are some tests that compare Antivirus prudcts against each other (AV-Test, AV comparatives), but the methodology is extremely flawed.
Problems with naive Antivirus tests
If a software detects a malware it does not mean it would've caused harm if undetected.
Alternatives to Antivirus software are not considered.
Antivirus software as a security risk is not considered.
None of these tests are with real users.
Questionable statistics
Bad statistics about IT security are common.
The most notorious example is probably CVE counting.
(see also Black Hat USA 2013 - Buying into the Bias: Why Vulnerability Statistics Suck)
IT security is largely not based on Scientific Evidence
Wait, aren't there plenty of scientific papers and conferences on IT security?
Most Cited Academic Security & Crypto Papers
- Candidate Indistinguishability Obfuscation and Functional Encryption for all Circuits
S Garg, C Gentry, S Halevi, M Raykova, A Sahai, B Waters, FOCS, 2013 - Candidate Multilinear Maps from Ideal Lattices.
S Garg, C Gentry, S Halevi, Eurocrypt, 2013 - FRESCO: Modular Composable Security Services for Software-Defined Networks.
S Shin, PA Porras, V Yegneswaran, MW Fong, G Gu, M Tyson, NDSS, 2013 - Homomorphic Encryption from Learning with Errors: Conceptually-Simpler, Asymptotically-Faster, Attribute-Based.
C Gentry, A Sahai, B Waters, Crypto, 2013
Papers that affect real software
-
DREBIN: Effective and Explainable Detection of Android Malware in Your Pocket.
D Arp, M Spreitzenbarth, M Hubner, H Gascon, K Rieck, NDSS, 2014 - Lucky Thirteen: Breaking the TLS and DTLS Record Protocols
NJ Al Fardan, KG Paterson, IEEE S&P, 2013
Not a single paper that tests with real users.
IT security and science
Most academic research in IT security is comparable to basic research.
Practical research tends to investigate interesting, but probably not very relevant parts of the problem.
Proposal for a randomized controlled trial on security software
Get a large group of users, randomly split them in groups:
- n groups using Security products.
- 1 group using "alternative treatment", e. g. automatic regular updates / application whitelisting.
- 1 group gets a user training ("Don't click on that attachment").
- Placebo (let them do whatever they did before).
Randomized controlled trial for security software
- Measure security incidents.
- Measure "side effects" (performance slowdowns, costs, downtimes, ...).
- Compare result after some amount of time.
"But this is really hard!"
Yes, it is. Sorry, science is hard.
Problems
- Ethics (situation comparable to medicine).
- How to measure incidents?
- Standard attacks versus targetted attacks?
Other things that could be tested
Safety of programming languages (e. g. Rust versus C++).
Application security (e. g. different browser brands).
An example that is both good and bad
FTC and passwords
The Federal Trade Commission (FTC) found out they had no scientific evidence for their recommendation to change passwords often.
FTC recommends the opposite
Regular mandatory password changes are probably not a good idea.
We have studies that say so.
However...
All of the studies are based on observational data, no intervention studies (Correlation != Causation). Nothing that comes close to a randomized controlled trial.
The studies measure things like password entropy, not real incidents (in medicine you would call that a surrogate endpoint).
FTC and passwords
Good: The FTC looked at the scientific evidence.
Not so good: The quality of the evidence was relatively low.
Caveats of Evidence-based IT-security
Future or very rare threats
Post-Quantum Cryptography: We want to protect against future attacks on cryptography.
Reproducible Builds: Protect against rare, but powerful attack scenario.
There are things you shouldn't study
Some IT security products make impossible claims.
"Full protection from malware" - that violates the halting problem.
Related debate in medicine: Should you study claims that violate the laws of physics? (Homeopathy)
Evidence-based IT-security
Today IT security is largely not based on scientific evidence - instead we rely on experience, expert advice or - even worse - marketing.
We should use Evidence-based IT-Security based on high-quality science. However the science largely doesn't exist.