As my friend Dennis Fisher noted on this blog yesterday, the 2016 Verizon Data Breach Investigations Report came out this week and, with it, a lot of follow-on reporting on the insights the report offered.

Make no mistake about it: Verizon’s DBIR (as it is known) is one of the best sources of information on the global security “landscape” that we have. Using broad strokes, it tells us a lot about the kinds of malicious activity that large organizations – Verizon’s customer base – are seeing. This year’s report, for example, highlighted the continued scourge of phishing attacks, credential theft, and attacks on web applications and point of sale systems as problems.

But if what you want in reading reports like the DBIR is something like a public health bulletin, an epidemiologic report or a warning, as you might get from the National Centers for Disease Control, you’re going to be disappointed. As good as the DBIR is, relative to its competitors, it still falls far short of that standard.

Rather, what we get from the DBIR is what we often get in the information security field: proof of whatever it is we were looking for. To put it simply: our reliance on private firms and privately funded security studies like the Verizon report leave our whole industry – not to mention our economy and society – vulnerable to what might be considered “confirmation bias,” the cognitive habit of looking for and preferring information (data) that confirms our previously held beliefs, hypotheses or preconceptions.

The security industry’s particular vulnerability to confirmation bias was underscored for me recently in a conversation I had with a good friend who is the CISO at a leading university. In addition to his many responsibilities, my friend had taken on the task of teaching a graduate level primer course in information security. As someone who is savvy about the industry, he gave his students an interesting assignment. He sent them home with an armful of threat reports generated by leading industry players – Cisco, Verizon, Symantec – and asked this students to report back on what they learned.

The result? His students came back with a lot of information, but not much understanding. Each report, they told him, seemed to present a different perspective on information security threats and risks. No surprise: that perspective tracked closely to the kind of service or product the company offered.

Now, admittedly, the Verizon DBIR makes a good faith effort to avoid that well-known pitfall. Over the years, Verizon has expanded its list of organizations contributing data to its report. This year, more than 60 third party security and technology firms collaborated on the report, including many prominent information security firms.

The data is impressive: the dataset for the 2016 report is made up of over 100,000 incidents, of which 3,141 were confirmed data breaches. Verizon’s analysis was based on a subset of 64,199 incidents and 2,260 breaches. Furthermore, Verizon recognizes the limits of its data. “We would never suggest that every last security event of 2015 is in this report,” Verizon wrote, acknowledging what it calls “sample bias.”

The problem is really with what is being measured and the approach to measuring it. Simply put: the Verizon report and others like it collect data from existing tools that were, in turn, created to address known problems. That makes it good for charting relative changes from year to year in those previously identified patterns and trends, but not very good at spotting new and emerging threats, or – frankly – at revealing the big picture of cyber risk, malicious activity and so on.

We can see that around the edges of the Verizon report, which pooh-poohs phenomenon including mobile attacks and risks stemming from The Internet of Things (IoT). “We still do not have significant real-world data on these technologies as the vector of attack on organizations.”

But even anecdotal evidence suggests that this gloss is misplaced. For one, there have been numerous reports of malicious activity linked to platforms that are properly considered IoT, including multiple reports linked to the Black Energy malware, as well as stories like this one about VNC Roulette, a site set up to expose vulnerable systems using Virtual Network Computing, a popular remote access technology used to manage industrial control systems. In Ukraine, a coordinated cyber attack on the electrical grid that involved attacks on SCADA and industrial control systems darkened 200,000 homes.

But SCADA and industrial control systems are not mentioned in the report, despite DHS’s ICS CERT being listed as a contributor. The grid attack in the Ukraine gets mentioned in a timeline of cyber incidents on the DBIR, but somehow the connection isn’t made to the larger concept of attacks on industrial equipment, critical infrastructure and the IoT.

This isn’t to criticize the data that is in the DBIR or the conclusions reached from that data. It’s merely to caution that, often, we tend to find what it is we’re looking for and, in doing so, we learn more slowly: impeding progress towards a cure.

This is a lesson that other disciplines learned at a high cost. Notably: in the 19th century, the medical establishment was convinced that the bacterial illness cholera was spread by “miasma” - or noxious fumes, like the fog that often blanketed London at the time. That theory prompted residents of infected areas, like Soho, to flee when the epidemic took hold – in hopes of escaping the noxious fumes.

It took physician John Snow to actually collect data on infections and deaths, interview residents of the Soho district where the outbreak took place and, eventually, identify contaminated drinking water from a water pump on Broad Street as the source of the outbreak. In short: Snow solved the problem by opening his mind about the problem, and using data to lead him to a cause in a place that nobody else was looking.

Don’t get me wrong, the Verizon report is a good read. It’s entertaining – deliberately so. But what is needed in information security isn’t an entertaining read. It is something much more like a public health initiative that relies on Snow’s epidemiologic approach: one that is comprehensive, that relies on objective and concrete scientific and statistical methods and, importantly, one that is freed from the commercial prerogatives of private and public companies.

My 2c.

Paul F. Roberts is the Editor in Chief of The Security Ledger and Founder of The Security of Things Forum.

John Snow image originally uploaded by Rsabbatini at English Wikipedia - [1] Originally from en.wikipedia; description page is/was here., CC BY 4.0, https://commons.wikimedia.org/w/index.php?curid=403227