The security scores on this web page are a supplement to the Orca Security State of Virtual Appliance Security which you can download here.

Research Methodology

This research was carried out using Orca Security’s SideScanning™ technology—a SaaS tool—usually used to scan an organization’s VM deployment en masse, including servers and virtual appliances. There is no need to install agents on the VMs or authenticate to them—neither of which are allowed by most virtual appliances. All that’s required is read-only access to the stored images.

Orca SideScanning uncovers a wide range of risks, including vulnerabilities, misconfigurations, weak authentication, the risk of lateral movement, active infections, and insecure data. Thousands of risks might be listed, which are then prioritized for the IT security team’s attention based on their deployment context.

For this research, Orca Security used SideScanning in a limited way to evaluate virtual appliances for only two things: vulnerabilities and to check if operating systems were up to date. No pre-configuration of virtual appliances was involved and there was no real operational context. The inherent limitations of using SideScanning in this way meant the product’s ability to detect operational weaknesses wasn’t used. Nevertheless, it let Orca Security quantify the security state of a wide range of virtual appliances and the diligence of the respective vendors.

Vendors had no prior knowledge of our research, and they ranged in size from the small and esoteric to the large and well-known. In each case Orca Security accessed the latest available revision of the virtual appliances and paid the least practical rate to do so. Where multiple versions existed, we tested each one.

Inclusion Criteria

During April and May Orca Security scanned 2,218 virtual appliances from 540 vendors, all products available for download from public marketplaces. The research was not selective, all source images were scanned. 63% (343) of the vendors had but one product, while 90% (490) had five or fewer. About 1% had more than 100, the most scanned images for a single vendor (Cognosys) being 293. These represented 13% of all the scans.

There were two grounds for not including a virtual appliance in the research. One was a prohibitively high cost, as a few vendors charge $100s per hour for access. The other was that a small number of products included unusual or highly customized operating systems, to a degree that general vulnerability databases do not apply.

Scoring Methodology

To quantify the results, each virtual appliance was assigned an overall security score between 0 – 100 based on five weighted parameters as shown below.The higher the CVSS scores of the vulnerabilities discovered the worst a virtual appliance was rated. There were 17 critical vulnerabilities selected for this research, the presence of which meant compromise was considered just a matter of time. All were high profile with well-known exploits such as Heartbleed, EternalBlue, and DirtyCOW.

Scoring Method for SideScanning™ Vulnerability Results



Criteria Scoring

(out of 100)

Summary Results (2,218 appliances)

Operating System in/out-of-date


100 pts. If currently supported and maintained, 0 if unsupported

11.4% (252) had an out-of-date operating system

17 Orca-defined critical vulnerabilities


25 pts. Deducted for each discovered; 4 or more = 0

46% (1,022) had at least one;

1.1% (24) scored 0

Vulnerability having CVSS score of 9 or more


5 pts. Deducted for each discovered; 20 or more = 0

61% (1,350) had such vulnerabilities;

1.6% (36) scored 0

Vulnerability having CVSS score between 7-9


1 pts. Deducted for each discovered; 100 or more = 0

92% (2,043) had such vulnerabilities;

10.5% (232) scored 0

Overall unique vulnerabilities


0.25 pts. Deducted for each discovered; 400 or more = 0

 (all vulnerabilities from above also counted here)

95.3% (2,114) had such vulnerabilities;

9.5% (212) scored 0

Scoring the security risk from virtual appliances

Software suppliers should make sure their products are well maintained and patches are provided as vulnerabilities are identified. This is not happening with many virtual appliances.

Each appliance tested was given a score between 0 and 100, based on criteria laid out above, the average overall score was 77.5. Based on its score each virtual appliance tested was given a grade as laid out in the table below.


Grades Applied for Given Security Scores


Min. Grade Score


No. of Virtual Appliances in Grade







Well Maintained




Above Average











Below 55





Vulnerabilities as a proxy for virtual appliance hygiene

Vulnerabilities are errors discovered after software has been released. In a worst-case scenario, exploit code might be created that targets a vulnerability and enables attacks. Software suppliers provide patches for known vulnerabilities and emergency patches when necessary. However, the window for hackers is often left open for longer than it needs to be because, even when fixes are available, those responsible for applying them fail to do so.

Counting known vulnerabilities and their severity is a good proxy in assessing the overall hygiene of a virtual appliance. While most vulnerabilities are not exploitable, the best practice is to apply patches as soon as they’re available to minimize risk in some future context. Failure to do so is indicative of poor practice. All of this equates to a virtual appliance bundled with hundreds of known vulnerabilities being a greater risk than one that ships with a few; it represents an ongoing risk to associated virtual machines (VMs) and other IT assets.

Vulnerabilities are classified using the industry-standard Common Vulnerability Scoring System (CVSS). CVSS assigns a severity score based on several metrics that include ease of use and possible impact, with scores ranging from 0 (no risk) to 10 (highest risk). Such scoring facilitates the prioritization of responses. The current CVSS v3.1 was released in June 2019.

A new vulnerability is also given a common vulnerabilities and exposures (CVE) number. For example, the Heartbleed exploit first seen in 2014 targets a vulnerability in the OpenSSL secure network socket and is known as CVE-2014-0160.

Virtual Appliance Security Scores

View in iframe below or open in new window.