VMware kernel debugging

Since I'm mostly travelling around between my different customers my laptop is the only computer I have around. To setup test environments for the different projects I use the free VMware player and WMware server products to create virtual machines, in which I can test different solutions.

One of my last projects was to find a memory leak in a kernel mode driver. To do this I needed to do some kernel mode debugging again, something that normaly involves two computers connected by a serial cable. But not any more. It's a simple thing to setup a kernel mode debugging session using VMware.

First we need to get VMware to export a COM serial port to the host. This can be done through a named pipe and some lines added to the VMware configuration file for the virtual machine.

serial0.present = "TRUE"
serial0.fileType = "pipe"
serial0.fileName = "\\.\pipe\com_1"
serial0.tryNoRxLoss = "TRUE"
serial0.pipe.endPoint = "server"

We also need to enable kernel mode debugging in the target OS. This is done by editing the c:\boot.ini file for the virtual machine OS. Start by copying the current startup line, and add the /debug, /debugport and /baudrate startup arguments. My boot.ini looks like this:

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Enterprise" /noexecute=optout /fastdetect
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Enterprise DEBUG" /noexecute=optout /fastdetect /debug /debugport=com1 /baudrate=115200


Now boot the virtual machine and select to boot into the DEBUG version. During the boot windows will stop and wait for the debugger. To start the debugger we give the arguments to connect to the named pipe that the VMware player now has created.

windbg.exe -b -k com:pipe,port=\\.\pipe\com_1,resets=0"

The boot will stop on a breakpoint during bootup, just enter g and press enter to continue.

To shorten the round-trip then doing driver development it's very nice to be able to change the driver directly on the target machine. This is since all bugs in kernel mode ends up in a BSOD (Blue Screen of Death). To do this a VMware utility called DiskMount comes very handy. With this utility it's possible to mount the virtual machine hard disk on your host computer and change the driver before the next boot.