Why Detection Rate is the Incorrect Metric

Blog

Why Detection Rate is the Incorrect Metric

Orca Security

Published:

Aug 28, 2019

Reading time:

4 Minutes

Detection rate is the first metric that comes to mind when discussing cybersecurity effectiveness. Everyone wants a higher detection rate – we’re naturally inclined to view a product that detects 99.7% of threats as superior to one that detects 99.5% of threats. Consequently, vendors place heavy emphasis on this metric with surface- level, impressive results. For example, more than half of the tested solutions in the recent av-comparatives test had a detection rate equal to or greater than 99.5%, however, these results are misleading and the focus is misguided.

Stop obsessing over fractions of a percent

It’s time to stop obsessing over a fraction of a percent of detection rate gaps and instead focus on gaps with higher orders of magnitude. When assessing your detection rates, you need to ask yourself the following: How many of my assets are actually protected by these solutions? Which attack vectors do they account for? Do they offer continuous protection or ‘point in time’ security?

The vast majority of breaches occur when assets aren’t fully protected by existing security solutions. We often find that assets are only partially covered (if at all), that not all vectors are accounted for, and that scans aren’t carried out continuously.

Learning from Research on 100s of cloud environments

Having conducted research on hundreds (thousands?) of cloud environments and the way their scanning is conducted today, it’s no wonder that so many company assets remain vulnerable to cyber attacks.

We’re no longer surprised to see the major risks other tools fail to detect, despite high detection rates reported. This is because the issues usually go beyond conventional detection rates- often due to the fact that these tools weren’t deployed across all points. This is why we need to shift organizational focus on effective detection rates.

Agent-based scanners, network scanners, and cloud security posture solutions (CSPMs) are commonly incapable of protecting cloud assets across all four layers of the cloud.

Here are a few examples of what we’ve seen so far:

Malware on production server – In this case, our first scan detected malware in the production environment. It sneaked in via an open-source tool that was in use. While the organization had tools in place to scan open-source packages in the CI/CD pipeline, the malware still managed to enter the system. The malware became well-known shortly after, and the organization’s lack of continuous scanning resulted in a very low effective detection rate.
Critical servers not patched for years – In another customer environment, we saw distinct differences between the assets that the IT department owns and manages. Some were maintained and patched regularly while others were left untouched for years. Some of them even ran OSs that had reached their end-of-support phase years before. Their vulnerability management tools were never integrated into the servers, leaving the IT department in the dark and the tools dangerously unmonitored. In this case, the per-asset integration required by the vulnerability management solution implemented resulted in a very low effective detection rate.
RCE on an internet-facing web server – Our customers in this instance used a vulnerable Apache struts server to host their website on an internal URL. The customer had a network-based vulnerability scanner that failed to detect this vulnerability as it never managed to reach the internal URL. The fact that it was dependent on crawling or manual configuration resulted in a very low effective detection rate, as it never reached the Apache struts server.

In all of these cases, which are only a small portion of many, the issue doesn’t lie in the detection rate of the organization’s existing tools, rather in their severely limited reach.

Call to action

Customers and security vendors alike must move towards a more real-life assessment of their tools. On top of the detection rate figure, they should count the effective detection rate in terms of the following: (1) the number of assets that will be protected; (2) for which attack vectors; and (3) whether it is a one time or continuous assessment. In simpler words, don’t forget to count the ‘footnotes’ of when a cybersecurity solution no longer applies to your environment, as they might be the doorways to future breaches.

The Orca Security Cloud Visibility Platform was built with a goal to maximize effective detection rate. This is one of the main virtues of Side Scanning – ability to reach 100% of the environment, without per-asset integration and using a one-time, read-only infrastructure level integration. In addition to providing obvious operational value, it means that each and every asset will be secured continuously and effectively.

Orca Security