DebugActiveProcess and DEBUG_ONLY_THIS_PROCESS

Then attaching to a process to debug it the default debugging flags is DEBUG_ONLY_THIS_PROCESS, meaning that if the process calls CreateProcess the child process will not be debugged. But if the process is started by the debugger the process can be started with DEBUG_PROCESS meaning that both the new process and any child processes created will be debugged by the same debugger.

I have been locking for a way to change this DEBUG_ONLY_THIS_PROCESS flag since Application Inspector is able to attach to a process and I want it to monitor all child processes as well.

A couple of days ago I stumbled on a comment on a Microsoft descrition on that was new in Windows XP, talking about Dynamic Control over Debug-child Flag. The description did not describe how to change the flag so I had to investigate.

After some debugging of windbg and dbgeng.dll, that can change the setting using the command .childdbg I found the kernel call to do that I want. It's the undocumented NtSetInformationProcess that is able to change the debug flags.

Here is the code to set the flag.

//
// Process options.
//

// Indicates that the debuggee process should be
// automatically detached when the debugger exits.
// A debugger can explicitly detach on exit or this
// flag can be set so that detach occurs regardless
// of how the debugger exits.
// This is only supported on some system versions.
#define DEBUG_PROCESS_DETACH_ON_EXIT    0x00000001
// Indicates that processes created by the current
// process should not be debugged.
// Modifying this flag is only supported on some
// system versions.
#define DEBUG_PROCESS_ONLY_THIS_PROCESS 0x00000002

typedef enum _PROCESSINFOCLASS {
  ProcessDebugFlags=31 // From ntddk.h
} PROCESSINFOCLASS;

typedef DWORD (CALLBACK * NTSETINFORMATIONPROCESS)(
  IN HANDLE               ProcessHandle,
  IN PROCESSINFOCLASS  ProcessInformationClass,
  IN PVOID                ProcessInformation,
  IN ULONG                ProcessInformationLength
);

static DWORD DebugSetProcessOptions(HANDLE hProcess,ULONG DebugFlags) {
  static NTSETINFORMATIONPROCESS g_NtSetInformationProcess = NULL;

  if (g_NtSetInformationProcess == NULL) {
    HINSTANCE  hNtDll;

    hNtDll = LoadLibrary(_T("ntdll.dll"));
    if (hNtDll == NULL) {
      return GetLastError();
    }

    g_NtSetInformationProcess = (NTSETINFORMATIONPROCESS)GetProcAddress(hNtDll,"NtSetInformationProcess");
    if (g_NtSetInformationProcess) {
      return GetLastError();
    }
  }
  
  return g_NtSetInformationProcess(hProcess,ProcessDebugFlags,&DebugFlags,sizeof(DebugFlags));
}

VMware kernel debugging

Since I'm mostly travelling around between my different customers my laptop is the only computer I have around. To setup test environments for the different projects I use the free VMware player and WMware server products to create virtual machines, in which I can test different solutions.

One of my last projects was to find a memory leak in a kernel mode driver. To do this I needed to do some kernel mode debugging again, something that normaly involves two computers connected by a serial cable. But not any more. It's a simple thing to setup a kernel mode debugging session using VMware.

First we need to get VMware to export a COM serial port to the host. This can be done through a named pipe and some lines added to the VMware configuration file for the virtual machine.

serial0.present = "TRUE"
serial0.fileType = "pipe"
serial0.fileName = "\\.\pipe\com_1"
serial0.tryNoRxLoss = "TRUE"
serial0.pipe.endPoint = "server"

We also need to enable kernel mode debugging in the target OS. This is done by editing the c:\boot.ini file for the virtual machine OS. Start by copying the current startup line, and add the /debug, /debugport and /baudrate startup arguments. My boot.ini looks like this:

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Enterprise" /noexecute=optout /fastdetect
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows Server 2003, Enterprise DEBUG" /noexecute=optout /fastdetect /debug /debugport=com1 /baudrate=115200


Now boot the virtual machine and select to boot into the DEBUG version. During the boot windows will stop and wait for the debugger. To start the debugger we give the arguments to connect to the named pipe that the VMware player now has created.

windbg.exe -b -k com:pipe,port=\\.\pipe\com_1,resets=0"

The boot will stop on a breakpoint during bootup, just enter g and press enter to continue.

To shorten the round-trip then doing driver development it's very nice to be able to change the driver directly on the target machine. This is since all bugs in kernel mode ends up in a BSOD (Blue Screen of Death). To do this a VMware utility called DiskMount comes very handy. With this utility it's possible to mount the virtual machine hard disk on your host computer and change the driver before the next boot.