Setup - VMM debugging using VMware's GDB stub and IDA Pro - Part 1

Sometimes you'll run into a situation that you can't analyze with a traditional kernel debugger like WinDbg. An example of such is trying to troubleshoot the runtime logic of PatchGuard (Microsoft's Kernel Patch Protection). In situations like this, you need to bust out the heavy tools. VMware has built in support for remote debugging of virtual machines running inside it through a GDB stub. IDA Pro, the defacto disassembler that most reverse engineers have, includes a GDB debugger. Together these make for a very powerful combo.

This article goes over how to setup VMware's GDB stub and how to connect to it using IDA Pro's GDB debugger.


  • A copy of VMware Workstation (free 30-day trial). I'll be using VMware Workstation 12.5.7 (build-5813279).
    • Unfortunately, VMware Player (entirely free for non-commercial use) does not expose the GDB stub interface.
    • You can use either the Linux or Windows build of VMware. I'll be using the 64-bit Windows build.
  • The IDA Pro application. I'm using IDA Pro x64 Version 6.95.160808.

Optional, but preferred

  • A Windows operating system installed on your host and guest (VM). These do not have to be the same versions of Windows. My host and guest OS are both running Windows x64 10.0.15063 (Version 1703).
    • This can be any OS supported by VMware such as Ubuntu. 
    • The second part of this tutorial (loading kernel symbols) assumes you're running a Windows 64-bit VM (AMD64).

Enabling the GDB stub within VMware

  1. Select the VM you wish to enable GDB stub debugging on within VMware.
    • VMs should be listed in the "Library" pane on the left of the GUI. If the "Library" pane is missing, you can restore it by selecting "View" then "Customize" and choosing "Library" (or hit F9).
  2. Ensure that the VM is currently not running. If it's currently active, power it off via the menu bar: "VM" then "Power" then "Shut Down Guest" (or Ctrl+E).
  3. Select "Edit virtual machine settings". Ensure that you are on the "Options" tab.
  4. Find the "Working directory" text field and copy the string to your clipboard. 'Cancel' out of the prompt.
  5. Go to the working directory.
  6. Right-click on the *.vmx file and "Open with" your favorite text editor. I'll be using Notepad++.
  7. Add one of the following lines to the end of the file, based on preference.
    • If your VM is 32-bit and you want to debug locally:
      debugStub.listen.guest32 = "TRUE"
    • If your VM is 64-bit and you want to debug locally:
      debugStub.listen.guest64 = "TRUE"
    • If your VM is 32-bit and you want to debug remotely:
      debugStub.listen.guest32.remote = "TRUE"
    • If your VM is 64-bit and you want to debug remotely:
      debugStub.listen.guest64.remote = "TRUE"
  8. The default port for the GDB stub is 8864 for 64-bit guests and 8832 for 32-bit guests. If you'd like to change what port the VMware GDB stub listens on (e.g. 55555), add one of the following lines to the file:
    • If your VM is 32-bit:
      debugStub.port.guest32 = "55555"
    • If your VM is 64-bit:
      debugStub.port.guest64 = "55555"
  9. If you want to start debugging immediately on BIOS load add one of the following lines to your file:
    • If your VM is 32-bit:
      monitor.debugOnStartGuest32 = "TRUE"
    • If your VM is 64-bit:
      monitor.debugOnStartGuest64 = "TRUE"
  10. To make it difficult to detect breakpoints that you've set using GDB, it's strongly recommended to add the following option too:
    • debugStub.hideBreakpoints = "TRUE"
    An important thing to note is that this option is restricted by the number of hardware breakpoints available to the processor (usually 4).
  11. Save the *.vmx file via "File" and then "Save" from the menu bar (or hit Control+S). Here's a copy of the contents of my *.vmx file:
    Close the file.
  12. Run the VM corresponding to the *.vmx file you just edited. Validate that the GDB stub is currently running by opening the vmware.log file in the same directory as the *.vmx file:
    If you see a message from "Debug stub" that tells you VMware is "listening" for a debug connection on a certain port number, you're in a good state.

    If you are missing that log line or have an error, ensure that your *.vmx file has the appropriate settings. Remember: you must edit the *.vmx file when the Virtual Machine is off or your changes may be lost.

Configuring the GDB debugger within IDA Pro

  1. Launch the 64-bit version of IDA Pro if you're debugging a 64-bit VM and the 32-bit version of IDA Pro if you're debugging a 32-bit VM.
  2. Skip the "Welcome" dialog (by hitting "Go") and go to the main disassembler window. Choose "Debugger" and then "Attach" and finally "Remote GDB debugger" from the menu bar.
  3. Enter the appropriate "Hostname" and "Port". These were set up by you during steps 7 and 8 of the "Enabling the GDB stub within VMware" section. Furthermore, these can be validated in the vmware.log file (this was done in step 12 of the same section). Then hit "Debug options".
  4. In the "Debugger setup" window, select "Set specific options".
  5. Ensure that the right "Processor" is set in the drop down box. If you're debugging a 64-bit edition of Windows (AMD64), select "Intel x64". If you're debugging a 32-bit edition of Windows (X86), select "Intel x86".
    Select 'OK' in the "GDB configuration" window. And then select 'OK' in the "Debugger setup" window. Finally, select 'OK' in the "Debug application setup" window.
  6. The VM will become suspended and a green "play" button will appear. At this point, IDA should bring up a window with the title "Choose process to attach to".
    Select "<attach to the process started on target>" and hit 'OK'.
  7. If you see this window, that means you're almost done.
  8. Select "Debugger" and then "Manual memory regions" from the menu bar.
  9. Inside of the "Manual memory regions" tab, right click and select "Insert" (or just press "Insert" on your keyboard).
  10. A new window will pop up. Enter in the "Start address" as 0 and the "End address" as -2. Make sure the right "segment" is selected (e.g. 64-bit segment for 64-bit VM debugging) and hit 'OK'.
    • This essentially maps all virtual memory from 0 to 0xFFFFFFFFFFFFFFFE (on x64).
    • "-1" is not an acceptable boundary for IDA Pro as the "End address".
  11. Find the "General Registers" window and find the IP register (RIP on x64, EIP on x86).
    • If the "General Registers" window is gone, select "Debugger" and then "Debugger windows" and finally "General Registers" from the menu bar.
  12. Right click on the IP register and select "Jump" from the context menu that appears.
  13. Your memory view will become synched to the IP register. If there are raw bytes listed and not code, don't panic. Place your cursor on the address of the IP and hit "C".
  14. Congratulations. At this point you've successfully set up VMware's GDB stub and IDA Pro's GDB debugger. You are now able to debug the VM and apply breakpoints through the IDA Pro GUI just as you would normally through a kernel debugger. Most of the functionality of the GDB debugger can be accessed through the "Debugger" menu bar.
    • This type of debugging is transparent to the kernel and therefore "debugger" checks like "KdDebuggerEnabled" and "KdDebuggerNotPresent" will not trigger. Furthermore, if the debugStub.hideBreakpoints option was enabled, breakpoints (up until the hardware maximum) will not make any inline code edits!

Final thoughts

Ultimately, the GDB debugger is not very useful without kernel symbols being loaded. One option, albeit a naive one, is to attach WinDbg as a kernel debugger while running IDA's GDB stub in the background. A tutorial on how to setup kernel debugging using WinDbg and VMware can be found here. You are then able to use the symbolic data that is provided from WinDbg to power debugging in IDA's GDB debugger. This is very cumbersome and has many disadvantages such as not being able to avoid kernel debugger checks.

Luckily, there is a better way. In the second part of this series, we'll discover how to load kernel symbols in IDA Pro's GDB debugger.


Popular posts from this blog

Setting up kernel debugging using WinDbg and VMware