Skip to Content

Test your setups virtually

Posted on

A key part of most setup development is testing from a known state to see that your setup leaves the machine in another known state. Usually, it’s from a “clean” state to one with your app installed but I’m sure I’m not alone with needing to duplicate bugs that come from a “dirty” machine. In both cases, you need to be able to quickly get a test machine to the starting state.

One way to get consistent machine states is with imaging software: Take an image of the machine in a particular state using software like Acronis True Image and restore the image before you begin each test. The downside to machine images is the time needed to restore an image; a bare operating system image might be only a couple of gigabytes for a svelte Windows XP image but your setup or app might have other prerequisites that balloon the image size. For example, my testing for Flight Simulator X Service Pack 1 and Flight Simulator X: Acceleration both require an installation of Flight Simulator X, all 13GB and 42,000 files worth. Such an image takes about 20 minutes to restore, not counting the reboots and loading the imaging software: Call it a good half an hour. With that overhead, I avoid re-imaging between install tests; the overhead’s not worth it. And surely I never author any bad setup or make even the slightest bug in my deferred custom actions, much less the rollback custom actions. Oh wait…

But virtual machines have significantly less overhead. Instead of re-imaging, you shut down the VM and discard changes made during the test session. Then restart the VM to start clean: The only overhead is the time it takes the VM to “power on” and boot. Most VM software lets you create snapshots to record the state of the VM at key points. You can also create new VMs as clones or deltas of other VMs, to save the time of installing OSes over and over.

There are two potential drawbacks to cloning VMs:

  1. Don’t forget your OS licensing.
  2. If you need to join your VMs to a network domain, you’ll have to use a tool like SysPrep or NewSID to join cloned VMs to the domain.

Of course, both those things are true with hardware machine images too. I avoid the problem by not joining VMs to domains. Most VM software offers other ways of getting test files to VMs, like shared folders that simulate a network share to the host computer or drag-and-drop.

VMs aren’t as fast as real hardware but unless you have a huge setup or want to investigate timing-related issues, it’s rarely a serious problem. Though it gives you a great excuse for a powerful dev box: A quad-core CPU with 4GB RAM and RAID 0 storage works really well. Fewer cores, less RAM, and non-RAID disks work too, of course, depending on how many simultaneous VMs you want to run.

VM software

Here’s a [partially obsolete and definitely out-of-date –Ed.] list of the VM software I’ve used.

  • Virtual PC, Microsoft’s desktop VM product. Free (as in beer). Easy to use, OK GUI. Biggest cons: It’s a single-threaded app, so multiple VMs don’t take advantage of multi-core CPUs and it has no automation API.
  • Virtual Server, Microsoft’s current server VM product. Free (as in beer). Multi-threaded with COM API. Biggest con: In the box, there’s only a Web administrative interface. VMRCplus, an unsupported WinForms UI, is available and while it has some rough edges, it’s a big improvement.
  • VMware Workstation, VMware’s desktop VM product. $189. Great multi-process GUI with automation APIs. Hardware support improves with each version; the current version, 6.0, supports USB 2.0. The only drawback is that because it’s not free software (beer or speech), you need to make sure you’re in compliance with license if you want to use it on multiple machines.
  • VirtualBox, innotek’s VM product. Free (beer and speech). VirtualBox comes in a closed-source edition that’s free of charge and a subset is available under the GPL. Multi-process and nicely automatable, via command line and XML configuration files. There’s a separate API that isn’t yet documented. The biggest downside is that virtual machines, hard drives, and CD/DVD images have to be registered so it’s slightly more complicated than the others to copy VMs among host machines. Secret setup geek benefit: The closed-source edition of VirtualBox comes with an installer – authored in WiX – that’s amazingly clean; it doesn’t quite pass validation but it’s very light on black-box custom actions.

Using machine images is vital if your setup requires particular hardware or if your app does and you need to test both the setup and the app. Otherwise, save yourself headaches, hassles, and horked dev machines and test your setups in virtual machines.