Kernel Debugging Vista Over USB (Or Not)

My Dell 380 at work has got just one COM port and I have been switching my serial cable between hosts as needed to get going. Since I had been hearing a lot about Vista supporting kernel debugging through USB, I thought yeah why not try it out. I could use some speed for sure.

I found an Intel document about it here and saw Hector’s memo on it here. I saw some evidence of folks getting nowhere in osronline forums so I figured, I may have some challenges ahead for this to work. No biggie.

I ordered Net20DC USB 2.0 Debug Cable¹ from PLX Technology, which is pretty much the only cable out there for this purpose and got it couple of days later. Since USB was originally designed for devices connecting to PC, and not for PC to PC connection, the debug cable poses as a device to the PCs at both ends and so it is different from regular USB cables in that way. Now, hubs are permitted on the host side (the machine on which debugger is running) and not permitted on target side (the machine being debugged) of the debug cable. So I was thinking of using a hub on the host side to potentially debug same host from multiple machines. I can still use my COM port on XP and older machines.

My windbg host machine on XP SP2 installed the driver once I plugged in the cable and device manager showed it as Ajays USB 2.0 Cable [or something like that] and everything looked fine. The driver that was picked was usb2dbg.sys [written by Microsoft].

I went ahead and plugged the cable onto my target Vista host. Vista could not find the drivers for it and gave me the option to search online. No luck there. Then Vista suggested I check if there is a solution to my problem. No solution there. Turns out you have to point Vista to usb2dbg.sys lying in your debugging tools installation folders [for example C:\Program Files\Debugging Tools for Windows\usb] and then everything is fine and dandy. Why did Microsoft decide not to ship this driver on Vista when it was in XP ? No idea.

In any case, after copying default Vista boot entry [via bcdedit /copy ] and turning on debugging on the copy [via bcdedit /dbgsettings usb targetname:name-you-want ], I downloaded USBView and started hunting for Port 1 on my target host, since that is a requirement for kernel debugging to work. Once I found out the port numbers and marked them such with a marker [so I dont have to do this again], I realized that the organization of USB ports on a machine can be rather disorderly with some ports missing in between. I did not open my box to check if there were internal ports that would explain the “missing” ports.

I made sure that both sides were working with EHCI Controllers and rebooted target and chose the debug boot option. Then fired up windbg and connected to the target name. It was not working. I was combing through the web to see if I was missing something, when I found the following in windbg documentation –

Additional software components need to be installed for USB 2.0 kernel debugging to work. For details, contact [email protected].

So I did, and I learnt that the EHCI Controllers have to implement KD support for this to work and it is optional for the Controllers. Also there is no programmatic way to know whether an EHCI Controller supports debugging, unless the BIOS exposes it and Microsoft is not aware of any Controller+BIOS combo that advertises debug support. Therefore there is no way to find out if debugging is supported on a system, other than by trial and error. If that sounds crazy already, check this out. Some vendors have debugging support in their controller, but have not exposed the port so you can be SOL on those systems !

I wish I had seen this and this before I bought the dongle. Now I am not looking forward to calling HP and Dell to find out whether their EHCI Controllers support debugging. Bottom line – chances of getting your kernel debugging setup through USB to work, are similar to your chances of winning the lotto.

I learnt my lesson and paid $97.35 ($83 + freight $7.50 + Sales Tax $6.85) for it.

¹net20dc seems to have vanished from PLX site. But I found net20dc being sold here.