Unrealistically Fast (or Negative) Ping Responses in Server 2003 Hyper-V Guests

I came across an interesting problem the other day while I was doing some unrelated troubleshooting on one of my Hyper-V guests.

The symptoms were that my Windows Server 2003 machine would return very strange results when pinging hosts, both internally and externally, such as returning all four responses within about half a second, yet measuring them at over 3000ms (which means they should have timed out, rather than given me a reading in milliseconds) as well as occasionally providing negative values for response times.

Obviously the results were completely inaccurate, but I couldn’t work out why the issue was only happening on a handful (not all) Hyper-V guests running Windows Server 2003 and none on Server 2008.

Turns out that this is an issue if all of the following are true:

  • You are running an operating system prior to Windows Vista or Windows Server 2008
  • You are running the current implementation of Microsoft Hyper-V (i.e. at the time of writing)
  • You have presented multiple processors to the Hyper-V guest

The issue occurs because the multiprocessor HAL in Hyper-V causes the guest’s operating system Time Stamp Counter (TSC) to skew. According to this blog the problem wouldn’t ordinarily occur if you were running Windows Server 2003 with SP2 unless the BIOS check fails to determine if the TSC should be used. More specifically, if I understand correctly the issue occurs because the processors (or cores, if we’re talking about a single multicore processor) are not in sync with each other, which produces sporadic out-of-time results where time sensitive operations (such as ping responses) are in use.

The resolution is to force the guest to use the PM timer instead of the TSC, by adding /USEPMTIMER in the boot.ini file and then restart. You can easily test this by running a ping -t to a host and checking for drastically abnormal results.

Migrating from Virtual Server 2005 to Windows Server 2008 with Hyper-V

I’ve recently migrated all of my Virtual Server 2005 machines to Windows Server 2008 with Hyper-V. The migration isn’t as smooth as it could be, but it’s not too difficult either.

Here are the steps that I took, keeping in mind that all of my guest operating systems are running Windows Server 2003 R2. I was going to take screenshots, but it’s been a long day.

  1. Firstly, you need to record the TCP/IP information from each of your VM’s to migrate, as you’ll need to reconfigure the NIC from scratch a bit later;
  2. When you’ve record the TCP/IP information, you should uninstall the Virtual Machine Additions from the guest operating systems – This is a fairly important step, as I’ll explain further on;
  3. Now you should shut down your guests to prepare for the migration;
  4. Using the Hyper-V Manager on your Windows Server 2008 machine, you should first configure the Hyper-V settings – You can do this by right clicking on the server in the Hyper-V Manager console, and clicking on Hyper-V Settings;
  5. I configured the Virtual Machines location to point to D:\Virtual Server, as the default is in an unusual location (like Virtual Server 2005);
  6. With your new path specified, you can create your first machine – On the right hand pane, click New -> Virtual Machine;
  7. Go through the options, naming your machine, allocating the appropriate RAM and assigning the NIC;
  8. When you get to the step to add a hard disk, select Attach a virtual hard disk later and then Finish;
  9. You should now have your machine visible in the Virtual Machines section;
  10. Use Explorer to open the location that you specified in step 5, and notice that Hyper-V has created a folder named “Virtual Machines” which contains a subfolder and an XML file with the same GUID (hexidecimal string) name;
  11. The XML file is the equivalent of the .vmc file from Virtual Server 2005 (where the machine configuration is stored), and the folder will be empty for now, but will be where the virtual machine’s running state (.vsv) will be stored;
  12. I recommend you create a folder underneath the D:\Virtual Server directory called Virtual Hard Disks and then create a subfolder for each of your machines under here – I copied the GUID name of the machine, to keep consistency;
  13. Locate your legacy Virtual Hard Disk (.vhd) file and copy it in to the appropriate Virtual Hard Disk subfolder;
  14. In the Hyper-V Manager console, right click on your machine and click Settings;
  15. Click on IDE Controller 0 and then click on Add on the right hand side, ensuring that Hard Drive is selected from the combo box above;
  16. Click on Browse and then locate your copied .vhd file;
  17. If you have additional disks, you can either add them on a virtual IDE controller, or a virtual SCSI controller (you need to add the SCSI controller via the Add Hardware option in the machine settings) – Note that Hyper-V does NOT support SCSI boot devices (I’m not sure why);
  18. Now you can start your VM by right clicking on it and selecting Start – Connect to the console by right clicking on it and selecting Connect;
  19. When it boots up and you log in, you’ll probably notice a few things – Firstly, if you are logged in via RDP you won’t be able to use the mouse to control the guest. This makes life pretty difficult, because of course your shortcut keys generally target to your local machine, or to your RDP window, depending on how you’ve configured your RDP client. This is why I recommend uninstall Virtual Machine Additions before moving your .vhd because navigating through Add/Remove is a bit tricky without a mouse and shortcut keys like ALT+TAB, etc… You should be able to follow the next steps without a mouse (note that if you need a mouse, you can either log on to the Hyper-V server locally, or you can install the Hyper-V Manager on a Vista machine, and if needed delegate access to remotely manage the machine – Third party remote server administration tools that don’t utilise the RDP protocol would likely work as well).
    The second thing you’ll probably notice, is that your machine is detecting new hardware – Escape out of these dialogues.
  20. Press CTRL+ALT+Left Arrow to lose focus to your virtual machine, and then click on Action -> Insert Integration Services Setup Disk (you can also just press CTRL+I);
  21. The setup disk will auto-run, and will ask you if you want to install the Integration Services – It will also prompt you to update the HAL;
  22. After you’ve confirmed the prompts and the tools are installed, your machine should reboot – When it comes back from a reboot, log in again and it should continue to install some more drivers – Note that you may also receive another “hardware detected” dialogue box which you’ll need to close before the drivers can finish installing (it seems to wait indefinitely otherwise);
  23. Your VM should reboot again, and at last you should have control of the mouse (I can’t remember if you get control in the last step, but you’ll definitely have it now);
  24. When you log in this time, you’ll probably notice that there’s a network error icon in the system tray complaining of limited connectivity – We need to remove the legacy NIC from Virtual Server 2005 before we configure the new NIC (well, we don’t need to, but Windows will complain that you’ve already allocated the IP if you use the same one);
  25. To remove this old NIC, run the following commands from the Command Prompt

    set devmgr_show_nonpresent_devices=1
    devmgmt.msc
  26. This will set a flag to allow devices that are not present to be displayed when hidden devices are unhidden in the Device Manager, and it will open Device Manager;
  27. In Device Manager, click on View -> Show hidden devices, and then expand Network Adapters;
  28. You should see your old NIC(s) which should be either a DEC 21140 or an Intel 21140 – Right click the NIC(s) and uninstall;
  29. Close Device Manager, and then run the following command to reset the non-present device flagset devmgr_show_nonpresent_devices=0
  30. Using the information recorded in step 1, reconfigure your TCP/IP configuration as normal.

That should be it – You should now have a machine happily living in your new Hyper-V environment.