Analytics Summary: VMware Security

Jun 27, 2008 (08:06 PM EDT)

Read the Original Article at

Our survey on the state of VMware security revealed some startling facts: Just four in 10 consider hyperjacking a realistic threat, and nearly half take a laissez-faire approach to virtual machine provisioning and management. Some even let business units deploy VMs with no oversight, perhaps because 20% assert that VMs are safer than physical servers.

The reality, and a concept that many IT and business managers fail to grasp, is that a virtual server is still a server. A production VM--and its host--must be held to the same level of rigor as a comparable physical production server, with identical change management policies for approval, deployment, patching, and other processes.

InformationWeek Reports

For now, accepted best practices are at least as important as VM-specific toolsets. Still, hypervisors must have security baked in from the beginning. Armies of attackers are no doubt working feverishly for the bragging rights that will come with being among the first to hyperjack--that is, to gain control over--a high-value physical server that hosts VMs.

So are industry-leading virtualization vendors doing enough to keep us safe? For example, will VMware's VMsafe program, which provides APIs with hooks into the ESX hypervisor, pay off for IT, or even help keep Microsoft's Hyper-V at bay?

Maybe, on both counts. Every security vendor we interviewed for our VMware Security Analytics Report, is focusing on product development for VMware. And every security vendor we interviewed for this report also has plans for Hyper-V or Xen product development. Making like Switzerland between VMware and Microsoft is a rational move, a reality backed up by our survey of 423 business technology professionals. VMware is still the dominant player in server virtualization, with 56% of installations, most of them Infrastructure 3/ESX. But our poll reflects the growing influence of Microsoft: 24% of respondents listed either Hyper-V or Virtual Server 2005 as their primary server virtualization platform.

This is far from typical estimates of 70% to 80% VMware ownership of the server virtualization landscape. An outlier? Perhaps. We expected Hyper-V to make a mark, but we must admit to being surprised by these results.

In our poll, when we asked about VM-specific security plans, 39% said they don't need specialized tools, opining that a VM is just another server.

Well, yes and no.

The problem is, the ease with which we can create and deploy virtual servers has gone to a few IT pros' heads--provisioning a virtual machine takes literally minutes. People who ought to know better are dispatching VMs into the wild at a pace that outstrips internal security review and audit procedures.

Blame it on budget pressure, customer demand, weak management toolsets, lack of VM-specific policies, the animal attraction of sexy technology, good old human foolishness, running out of data center space, or any combination of the above. The fact is, many organizations today are running ESX shops by the seat of their pants.

And we don't expect Hyper-V to help matters, but that's another article.

An ESX host, at its most basic conceptual model, is a single physical server running multiple virtual servers, also known as guests or virtual machines. The host platform does this by essentially tricking each VM into believing it has sole access to all physical system resources. Operating systems like to think that they have exclusive one-on-one relationships with system memory, CPUs, I/O calls, and other nuts-and-bolts connectivity. To get around this, modern hypervisors operate between guest machines and the physical host, intercepting calls to hardware and delegating physical resources to the VMs as required. An ESX host becomes a self-contained environment, virtualizing servers' access to RAM, disk storage, and the network.

And therein lies the risk: This self-contained bundle of servers and virtual networks is outside the sight of all those expensive intrusion-detection and network access control systems, active traffic analyzers, and any other tool in your security kit--all will view an ESX host as a single physical server.

Moreover, intrahost links, hypervisor-to-VM connections, and the virtual network likewise will be invisible to traditional tools. Take patching, among the most fundamental of server security functions. When we asked how distribution and installation of security patches is handled, 15% said "we do the best we can."

Virtualization security risks fall into two primary categories: intrahost threats and hyperjacking.

It's intrahost threats that make the security folks we spoke with most nervous. The primary concern is that a guest OS could be compromised via a known OS or application exploit, then other VMs on the same host could be targeted from the compromised VM. With all attack vectors occurring within the virtual network on the physical ESX host, conventional network security tools would fail to detect suspicious traffic, as long as attackers contain their actions to VMs on the same host as the compromised guest.

chart: In your organization how are VMs viewed in terms of security?

If an intrahost attack is the more likely exploit, the worst-case theoretical scenario in a virtualized environment is hyperjacking, where an exploit leads to a compromised physical host server, allowing criminals full access to all guests on a given machine. In subverting the hypervisor, malicious software could disguise its presence from security tools that may reside not only on the physical network outside the host, but even within hosted-OS partitions or in any software layer above the hypervisor. The exploit situation is analogous to the threat of cloaked rootkits compromising a standalone server OS: If you successfully control the hypervisor, you can manipulate all packets traversing the hypervisor and are in a position to sample, redirect, or spoof any data traffic to or from hosted VMs. Lacking a security provision, guest operating systems have no way of knowing they're running on a compromised platform.

Hyperjacking has been widely discussed for at least a year, and yet, incredibly, 36% of respondents to our reader survey had never heard the term. Just 8% are concerned that hyperjacking keeps their organizations from storing sensitive data on VMs.

We see why there's concern among security experts--when you look at large-scale virtualization systems that enable 10, 50, even hundreds of hosted servers to run on a single hardware platform, the potential risk for loss of control and revenue is enormous. It's in everyone's best interests to maintain the integrity of the hypervisor while building in multiple fail-safes for hosted operating systems, to ensure that they're communicating with an untainted hypervisor as a bridge to the underlying hardware and external connections.

Moreover, bad news sells. Gartner made headlines predicting that a patch-worthy hypervisor vulnerability will be discovered in a mainstream product before the end of this year.

Virtualization and security vendors tend to agree that hyperjacking is a slim possibility but recommend that we take a rational risk analysis and return-on-security-investment approach toward allocating security dollars.

We couldn't agree more. Applying a healthy dose of common sense to potential threats will yield the best return on your security investment, while reinforcing your legacy security infrastructure and maximizing staffing. Consider your current risk profile and potential zero-day exposure as you follow conventional change-control and security procedures for your VM hosts and guests. Frankly, it doesn't matter if your company is looking at VMware, Microsoft's Hyper-V, Citrix's Xen offerings, or one of the smaller players in the market to implement virtualization. While virtualization-specific concerns exist, shoring up traditional data and infrastructure security will still yield a bigger short-term payoff versus large investments in fledgling virtualization-specific security offerings.

In February, VMware announced an official API for ESX hypervisor security under its VMsafe program, aiming to provide a common toolset for monitoring traffic through the hypervisor, making hosts a touch more transparent. The list of backers reads like a who's who of security vendors and includes everyone from big guns to virtualization security specialists.

VMsafe enables third-party security products to gain visibility, using ties to the ESX hypervisor, into the operation of a virtual machine to identify and eliminate malware, such as viruses, Trojans, and key-loggers. Security vendors can leverage VMsafe to detect and eliminate malware that is undetectable on physical machines.

VMware's pitch implies that VMs running in a VMsafe ESX host are actually safer than physical servers because of process monitoring and inspection techniques that can occur, thanks to the abstraction of guest operating systems. Visibility into hypervisor activity provides the opportunity for revolutionary observation and analysis tools for security vendors.

VMware says VMsafe includes sharing an "open, interoperable, and cross-platform set of technologies with partners so they can provide innovative security solutions." "Open" in this case refers to compatibility of products within the VMware ecosystem; applications from diverse third-party vendors should play well with others, as long as all apps are written to spec. VMsafe should provide customers with better security, granularity, visibility, correlation, and scalability in virtual machine deployments. VMsafe integration allows third-party security tools to monitor VM memory pages and CPU states; enables filtering of packets within the virtual network, both to the hypervisor and intrahost connections between VMs; provides in-guest, in-process APIs for complete monitoring and control of process execution; and allows for storage control for guest VM disk files.

Current VMsafe partners are Altor Networks, Apani, BigFix, BlueLane, Catbird, Cenzic, Check Point, Configuresoft, F5, Fortinet, Fortisphere, IBM, Imperva, Kaspersky Lab, McAfee, Montego Networks, Reflex Security, RSA, Secure Computing, Shavlik, Solidcore, Sophos, Symantec, Third Brigade, and Trend Micro.

So just how safe is VMsafe going to make us? Time will tell. While a number of capable virtualization security products already are on the market, and most of those vendors are VMsafe partners, as of early June, VMware and its partner companies had yet to produce a security product to the VMsafe specifications.

Still, we should see something soon. In every case, interviews with representatives from VMware, Symantec, Fortisphere, and Reflex Security showed a strong, almost obsessive, drive to get products to market.

It's encouraging that VMware is partnering with a wide range of companies, from small startups with niche, virtualization-centric security products to large, established vendors. We've been critical over the past few years concerning the absence of VM integration with enterprise-level security tools--and the seeming apathy of vendors in this regard. In hindsight, it's clear that established security players such as McAfee, Symantec, and Sophos were reluctant to invest heavily in potentially proprietary hooks for securing VMware; VMsafe provides vendors with a standard API, and customers with a level of confidence for investment. Still, 30% of respondents to our survey say they won't make any virtualization-security-specific purchase in fiscal year 2009. Fifty-eight percent will make only minor investments in virtualization security, planning to spend less than $50,000 on VM-specific tools over the next year.

That sounds about right to us. Unless you have a clearly identified risk associated with your ESX environments, we recommend holding off on making large-scale virtualization-specific security investments. A fiscally conservative and targeted approach makes sense until VMsafe is fully baked.