Whatever Happened To ...: March 2007 Archives

My original coverage of Vista's backup utility was just a walk-through of its features, but as time has gone on it's become clear to me that many of the people who used the original NTBACKUP utility (or a third-party backup application) are really not happy with the new Vista backup tool.  They find it too simple (shilling for simpleminded), and want to have more control over exactly what gets backed up and where to.  In this episode of "What Happened To...?" I'll address these problems and suggest some workarounds.

1. What's the Vista Backup Utility and What's the Problem with It?

In an earlier article (linked above), I wrote about the Windows Vista Backup Utility, which is designed to back up documents and data to an external device (a hard drive, CD-ROM or network location), either on demand or on a schedule.

The single biggest problem that most people have with Vista Backup is the lack of flexibility in terms of what is backed up and from where.  In the original NTBACKUP tool, you could choose exactly what files and directories were backed up, and where to back them up to:

In Vista Backup, you can only choose one or more drives to back up from, and what TYPES of files from each you want to back up:

This creates some serious problems, since it's now very difficult if not outright impossible to include some specific file types and exclude others.  For instance, the "Compressed files" option includes .ZIP and .CAB files -- which I do want to back up -- but also includes .VHD (Virtual PC's Virtual Hard Disk format) files, which I don't want to bother with.  There's no way to include one but exclude the other, so I'm stuck backing up both of them unless I move the .VHD files to another drive (which I really don't want to have to do!).

So the biggest problem with the Vista backup tool is its relative inflexibility.  It's been made much easier to use, but at the cost of being that much harder to manage.

2. How Do I Even Know What's Been Backed Up?

Another complaint I've heard about the new backup tools is that it's too hard to tell if a given file has been backed up, as there doesn't seem to be a central manifest of backed-up files.

The best way to find out if a given file has been backed up is to right-click on the file, select Properties | Previous Versions, and wait for the File Versions list to populate.  There's no visual feedback that indicates if the list is still populating itself, so you may need to sit tight for a bit before everything appears.

 The "Location" column will tell you where previous versions of the file reside.  If you see a "Backup" entry for the last backup date (which is visible in the main window of the Backup and Restore Center, then that file's been backed up.  (If the "Location" column reads "Shadow copy" for a particular revision, that means that revision is stored in that volume's shadow copy repository.)

3. What's To Be Done?

There's a few things you can do in lieu of using the regular backup utility.

1. Use NTBACKUP once more

After Vista was released to the public, Daniel Petri did some research and found a way to get NTBACKUP running in Windows Vista.  It involves obtaining a copy of NTBACKUP and some support files from an existing copy of Windows XP, and then activating the Storage Manager service in Vista (which is not normally running by default).  If you're familiar with NTBACKUP, or already have backup files in the NTBACKUP format, you can use it more or less as you did in XP and Windows 2000.  You should not use it to make or restore System State backups, though, and (as with many legacy apps) NTBACKUP runs best in admin mode.

Many people also complained that people couldn't restore backups made with the classic NTBACKUP program.  To that end, Microsoft decided to release a free download for all those who had a licensed copy of Windows, which allows you to restore NTBACKUP repositories in Vista without needing NTBACKUP itself.

One major disadvantage of using NTBACKUP is that the files generated by NTBACKUP cannot be restored by the Vista recovery disk; you have to restore them by using NTBACKUP itself or with the above utility.  Also, you can't perform full system backup and restore with it as per the Vista backup utility.  Finally, NTBACKUP requires the Storage Manager service to be installed and enabled (there are instructions on how to do this within the download).

2. Use a third-party Vista-aware backup program.

If you don't want to use NTBACKUP, you can always turn to a third-party backup solution, as long as it's specifically marked as being Vista compatible.  Unless the app is Vista-aware, there will be a marked lack of integration with Vista (no backup notification, etc.), and backup sets may not be generated or restored correctly.

3. Change your storage habits

This is the least appealing option, and the most cumbersome.  If you have more than one drive, you can move everything that you don't want to have covered by the backup utility to another drive, and let Vista's backup automatically handle what's in your Users folder.  Again, though, this is probably not going to sit well with people unless they had planned on breaking down and remodeling their work habits in the first place.

In this installment of Whatever Happened To ... ?, I'll be talking about something that I touched on before in the past -- the way Windows Vista's defragmentation application has been simplified rather drastically.  There are some possible ameliorations, however.

[Note: this entry contains some previously published information in a new form.  If you've read the earlier piece I did on this subject, I encourage you to revisit this one.]

1. What's the Defragmentation App?

In Windows XP, the Defrag app was either the command-line or GUI application that allowed you to defragment the computer on demand.  It was also possible to set up the Task Scheduler to run the command-line version of Defrag on a regular basis.

2. What Happened to the Defragmentation App?

In Windows Vista, the Defrag GUI was replaced with something completely different:

You'll notice that one of the things now missing is any kind of statistics or progress bar.  This caused a great deal of annoyance among experts, who felt they were being deprived of valuable information.  What's the point of having a tool like this if we can't even see the progress meter?

Here is why this issue is not as obvious as it might seem:

1. Automatic scheduling

One of the things that has been added is an automatically scheduled defragmentation job that runs once a week.  In other words, a user who runs Vista and knows nothing about defragmentation will have their drive defragmented for them in the background during periods of low activity.  It means one less thing for everyone, expert and amateur alike to worry about.

2. Generic defragmentation statistics are not always useful

Second, the more I learn about fragmentation on a drive, the more I realize there is no one convenient metric that you can use to describe how badly a drive is fragmented.  You could talk about individual file fragments, but what if they're for files that are rarely accessed?  Or free space fragmentation, but what if the free space "fragments" are so large (50MB or more) that it's not going to create an issue for any new file written in it?

I guess it might be possible to come up with some kind of global "fragmentation index" -- like the way many of the newer anti-malware programs have some kind of "threat index" or "system protection index", but in the end I'm not sure that gives us any more useful information than we had before.  If a drive is fragmented badly enough that it's impacting performance, then we tend to take a fire-and-forget approach to the whole thing: we defrag the drive and move on to doing more serious work.

3. Defrag progress statistics can also be misleading

But what about the progress display?  Here's a list of the reasons for eliminating the display (and the lack of any progress data), straight from the horse's mouth:

The new interface seems “dumbed down.” Why remove all the detail?
...one of the biggest and consistent complaints we had from users (broad sample here from home users to experienced IT Pros) in the past was that a vast majority of them had no idea what the detailed fragmentation statistics they saw meant. The Windows XP graphical view also had some limitations and inaccuracies that prevented it from being included in Windows Vista. If you really want to keep a close eye on fragmentation, I’d recommend using the command-line tool Defrag.exe. [Emphasis mine]

Why was the defrag progress indicator removed?
Part of the problem with the Windows XP defrag tool was that percent complete was not accurate or meaningful. Depending on the phase of defrag, 1% of progress could take from several seconds to minutes, which made the progress indicator highly unreliable. The difficulty here is that since defrag is a multi-pass process (multiple iterations of file defragmentation and free space consolidation) there is no way to accurately predict when defrag will complete since the number of loop iterations and how long each takes are highly dependent on the layout of the files on the volume, the level of file and free space fragmentation, and the other system activity. While I agree that having no progress is bad, misleading progress I believe is worse. Also, the idea behind the new automated defrag is that users will not have to think about it not worry about the progress it is making. With defrag running regularly, the system will be close to optimal levels of fragmentation, and subsequent defrag runs should not take long. [Emphasis mine]

I should note that there has been a significant change to the way the defragmenter "sees" fragments on NTFS volumes.  Any given file fragment that is larger than 64MB is not consolidated by default -- it's left as-is.  This is because a fragment that large is probably something that's read sequentially anyway, so it probably won't appreciably improve the performance of the file if you forcibly collect all of its fragments and line them up.  (A file that large also doesn't tend to get read all the way through very often.)

3. What's To Be Done?

1. Leave it alone

This sounds like blunt, almost know-nothing-ish advice, I know.  But from what I have seen it's better to simply let Vista deal with fragmentation in its own way and at its own pace than to try and micromanage the whole thing (and with questionable performance returns).  In the long run, your time will be better spent on other things.

2. Use the CLI defrag

If you must defrag manually, use the CLI defrag command.  For a really aggressive cleanup, run it once a month with the following options, preferably before you go to bed:

defrag c: -v -w

This produces a verbose defragmentation report and forces a full consolidation of files.  It's actually useful to do this if you just upgraded from an earlier version of Windows and the drive is a mess of fractured directories, but in my opinion the full defrag is a waste of time if you do it more than once a month.

If you want to defragment with some kind of visual feedback, you can schedule the CLI version to run with a window [how?]

If you just want a fragmentation analysis, run defrag c: -a -v, which returns detailed statistics about the drive.  The results are not graphical, but in my opinion they are more useful than looking at a graphical map.  Again, a map can be misleading, since it may not give you a valid picture of what actual impact the fragmentation has on the drive.  The stats in the analysis includes things like the total available free space or the single largest contiguous free space on the drive, two things that are immensely important as far as fragmentation goes; the less free space you have, the more of a problem fragmentation is going to be and the harder it will be to defragment in the first place.

3. Use CONTIG to defragment individual files

If you want to defragment specific files, pick up the Contig utility from Sysinternals.  It's a little tool which defragments a single file through the Windows file-system APIs.  Contig is also used by the Power Defrgamenter utility, which can be used to defragment a single file, a whole directory, or a whole drive by using Contig.  It's most useful if you have a large file in several fragments that are not being defragged conventionally, but you don't want to go through the time and expense of running defrag with the -v switch to clean it up.  (Note that you'll probably need to run these programs as admin for them to work properly.)

4. Use a third-party, Vista-aware defragmentation app

I am putting this option last simply because I've grown less personally convinced that third-party defragmentation apps are crucial in a desktop environment.  The situation with servers is different, however; there's more than a few scenarios where a server can benefit from having some kind of fairly aggressive defragmentation.  (This is a topic that deserves its own discussion so I won't go into it here.)

Most of the commercial defrag programs have been fairly well-documented, like Diskeeper, but there are a few free ones that are worth mentioning -- JKDefrag, for instance, which does have a progress display and a great many other useful features besides.  One flaw: as of its current version it does not leave alone files tagged for prefetch, but the author is aware of this and may be including provisions for it in future releases.