As you have probably noticed, a virtual machine is more than just a CPU emulator running as a process on a host machine. Modern virtual machines, such as VMWare also provide support for virtual various hardware as well as some means of interaction between host and guest machine. For example, VMWare provides clipboard synchronization, mouse synchronization and other similar functions. It is evident that such a mechanism can be adapted to exchange arbitrary information between guest and host.
Let's make a little digression and see, in which ways can such communication between guest and host be possible:
The shared memory approach should be the most fast, however it has a very significant disadvantage: synchronization. To ensure maximum performance, some thread should continuously check some status bit to determine whether new requests are issued from the other side. This will mean 100% CPU load for all the time, that is not very good. Moreover, Shared memory API was an experimental feature in VMWare Workstation 6.0.x and is deprecated since 6.5. The VMCI Sockets API that appeared in 6.5 is reported to be very slow and unreliable. Another problem is that VMCI API works in user mode, however, to implement a kernel debugging extension, the communication interface should work directly from kernel avoiding any library and OS calls.
The host-to-guest requests can be theoretically implemented in an interrupt-like way: a host calls a function forcing guest machine to execute some kind of an interrupt handler receiving request parameters and returning some data. However, no such interface is documented and no evidence of it actually existing was found. Moreover, it would not be very convenient to use such model, as Windows Kernel calls KD extension functions to send or receive a packet and does not specify any calbacks.
The guest-to-host requests can be most useful, as such an interface could be used for marshalling KdSendPacket()/KdReceivePacket() calls to make them being executed at host side. VMWare contains such an interface called VMWare backdoor. Let's see how it works in detail.
When you install VMWare tools, some usability-related features are added to the VM. For example, mouse cursor synchronization. Let's see how it is implemented:
To get some information from host, VMWare tools package performs a so-called backdoor operation. A partial description of backdoor operation types can be found on this page. Another way to find information about VMWare backdoor interface is to look at open-vm-tools project source code. The following function declared in backdoor.h actually perform a backdoor operation:
The following definitions from backdoor_def.h show the possible backdoor request types:
Let's now make a simple program querying the current mouse cursor location using VMWare backdoor interface:
If this program is executed on a real machine, the in instruction will cause a "privileged instruction" exception, as user-mode code runs in Ring 3. However, when this program is executed on the virtual machine, it will print the correct mouse cursor position.
One of the methods of executing guest-to-host requests, that was described above, is using Backdoor interface. However, patching VMWare code to support some additional backdoor commands could be tricky. For that purpose VMWare provides more flexible interface called GuestRPC. Basically, a single GuestRPC call consists of a sequence of requests:
The following functions are used in open-vm-tools to perform GuestRPC calls:
GuestRPC commands can be executed using the vmware-service.exe tool from the VMWare Tools package. For example, the following command prints the guest IP address:
KDVMWare provides a convenient object-oriented way for executing GuestRPC commands: VMWareRPCChannel and BufferedRPCChannel classes. Here is an example of a simple C++ program querying guest machine IP:
By using VMWareRPCChannel class it is possible to execute arbitrary GuestRPC requests, that VMWare supports. However, the question of adding our own request types is still open. Let's examine the VMWARE-VMX.EXE internals. When a GuestRPC is being issued by guest, code inside the VMWARE-VMX.EXE searches the so-called GuestRPC handler table for a handler corresponding the the issued request. A GuestRPC handler entry format can be defined by the following structure:
The code for fetching and executing a handler from a table can look like this:
The GuestRPC table is contained in a data section of the VMWARE-VMX.EXE process, i.e. it is not allocated from heap. However, initially the table is filled with zeroes. All entries are being initialized on VMWare startup. By setting an unused entry from the table to our custom handler, it is possible to execute our code on a request from guest OS, passing some parameters to it and getting some data back. The last problem on the way to adding our own GuestRPC handler to VMWare is about finding the table in memory.
As the table is not exported from the EXE file, it should be manually located using some sort of an algorithm. The algorithm used in KDVMWare is relatively simple:
In order to speed up table patching, KDVMWare maintains a database of VMWARE-VMX.EXE versions. For each version (identified by file size, time stamp from PE header, and first PE section size) the offset and size of GuestRPC handler table is stored. When KDCLIENT.DLL is loaded inside VMWARE-VMX.EXE, it tries to find a corresponding entry in the database, and in case of failure runs the detection algorithm.
To see, how the GuestRPC mechanism is used by KDVMWare, check out this page.