As a long-time CISO, I’ve been on the receiving end of … a lot of vendor sales pitches. So much so that I created a quick template to respond to all of those unsolicited messages. For the most part, vendors would either quietly disappear, or reply with good grace (for many sales development representatives, even being acknowledged as having a difficult job was a positive improvement). But sometimes, the responses would get a bit combative, usually with something like, “Don’t you care about your security? Don’t you know my solution will improve your security?” Let’s ignore that uncharitable first question, and focus on the second.
For any given vendor, are they likely to improve my security?
Yes, they will. Maybe by a small amount, maybe by giant leaps. Almost every vendor’s solution is going to make you more secure. The most challenging task a CISO faces isn’t spotting the handful of snake oil vendors (although that might be the most fun). It’s spotting the vendors who might make you a bit safer, but whose hidden costs are so massive that they’ll not only derail their own deployment, they’ll likely throw your entire security program off course.
Hidden Security Cost 1: Agents
A really large number of security products rely on the installation of an agent: a small bit of software that needs to be installed on every system to provide protections on that system. It sounds minor, trivial even, but it can be anything but.
If you have a completely homogenous environment – every system is identical to every other system – then agent integration is probably not a major issue. Most environments, however, are more heterogeneous. Windows servers sit side-by-side with Linux servers (and the Linux servers come in a whole rainbow of different distributions), while applications are built in every environment known to humanity, and a few more we haven’t heard of … yet. An agent has to be able to safely operate in each and every one of those environments to provide full coverage; and if anything goes wrong during the deployment phase, your agent will be blamed, rightfully or not. And then disabled as part of the incident response, and you’ll have to start arguing after the incident to have it turned back on.
Agents also use resources on the machine. Ask your vendor for the performance overhead, and you’ll probably get a vague non-answer like, “oh, not very much, just a few percent.” A few percent of what, you might ask? Often, agents are benchmarked on idle systems, and CPU usage might be measured. But what often matters is how the agent interacts with the system at peak load. Most enterprise users probably have a horror story about a laptop locking up for an antivirus scan right as they are headed into an important presentation; imagine the same concept happening to your webserver during a Black Friday sale.
Widespread agents are also a vector for supply chain attacks. The Solarwinds breach is one example, where an IT management agent, installed everywhere, created an avenue for compromise. The security risks of our own tools is a cost that most security teams don’t consider, but ought to be included in any evaluation.
Hidden Security Cost 2: Inscrutable Alerts
Anyone that has naively used a new security product that aims to find issues in their environment has encountered a flood of inscrutable alerts. Why does it matter that a machine replies to ICMP timestamp requests? (HINT: it really doesn’t). The challenge with alert definitions in security products is that the incentives are entirely misaligned between the vendor and the buyer. The vendor doesn’t want to have the dreaded false negative – hiding an alert that is actually important – while the buyer is focused on the opposite problem – trying to avoid the meaningless alerts that don’t bring actionable value.
That actionable value in an alert is really hard to define in advance. Sometimes, it’s contextual. Perhaps an adversary can read all the files on a web server. Is that a problem? It might be; that server could have credentials to your production database. Or it might be much less interesting, if the server only handles a stock brochureware site.
For many security teams, they aren’t even the true consumer of the alerts. Alerts get passed on to the business unit, and the security team experiences the pain of being ignored. Often, that’s a reasonable reaction from a business team that gets handed a thousand action items in one fell swoop. Many organizations end up in an uncomfortable situation where the devops team is demanding prioritization while the security team is demanding action. While both teams have a point, the valuable security work isn’t getting done.
At the heart of the problem is a mismatch between who the product is implicitly designed for – a deeply technical security architect and operator with excellent project management skills – and who actually ends up using it – a junior security analyst or project manager who is growing by leaps and bounds every day. The ability of a deep security professional to relatively quickly prioritize alerts would be handy, but that’s an expensive skill set to throw at thousands of alerts. Unfortunately, most tools don’t have the architectural or organizational context to provide a first pass at prioritization to aid the analysts in quickly driving positive security change.
Hidden Security Cost 3: Complex Deployments and Organizational Friction
Whenever a complex project has to be rolled out to multiple organizations, no organization really wants to be first; because the first mover bears the burden of issues in integration. They likely also will need to redo the rollout, because the company will have learned lessons along the way about a better implementation than the one first proposed.
Most project managers have experienced this pain firsthand: peers wanting to see someone else’s success before they do the bare minimum, surprising roadblocks and speed bumps derailing a project that was already moving extremely slowly. And once a project has been moving too slowly for too long, it becomes implicitly deprioritized (“well, since we didn’t do it the last three years, why is it now important enough to do this year?”).
The organizational coordination cost increases supralinearly as a function of the technical cost: the more work a team will actually have to do, the more pain they’ll add in coordinating getting that work done. It’s not even vindictive or retaliatory; if an organization has too much work on its plate, it needs to push back on large or nebulously defined workloads.
Ending up as Shelfware
Many security products end up as shelfware, or partial implementations, because their rollouts fell afoul of one or more of these pitfalls. And executives may not even realize that the projects have had limited or no success, because they’re still writing checks to the vendor for a solution that isn’t rolled out or is being ignored.
Curious to see how Orca Security avoids these pitfalls? Read more in TAG Cyber’s report: Cloud Security and Compliance: Vendor Landscape and TCO Analysis.
