March 2007 Archives

When Vista is set up, the disk defragmentation task is set to automatically defrag all the drives in the system.  This isn't a bad thing, but what if you want to get a little more control over the defrag process -- for instance, defrag one drive at full speed on one day, and defrag another drive in idle speed on another day?

In this little tutorial I'll show you exactly how to do that in Vista, which will also serve as a quick primer in how to create a scheduled task in Vista.  The process of creating a scheduled task is a little different than it used to be, but there are that many more options that can be set because of this.

Mark Russinovich has posted the third installment of his ongoing series about the changes to the Windows Vista Kernel.  This installment focuses on the Kernel Transaction Manager, how crashes are handled, and a number of security and reliability features like protected processes, Address Space Layout Randomization, BitLocker and Volume Shadow Copy.  It's more of the goodies that I've come to expect from Mark, and if you're remotely curious about how Vista has been changed under the hood I recommend you read the whole series.

No, this is not an April Fools' gag come early.  Microsoft's venerable database solution FoxPro is approaching the end of its supported lifespan (I posted about this earlier), but many people were worried it would simply disapper completely.  As it turns out, MS has decided to open-source the core of FoxPro, although they're no longer working on future versions of it.

One of the reasons that's been cited for FoxPro's incredible customer loyalty is the fact that it is in a number of embedded systems (and that it was a great piece of work for its time, to boot).  The author of the linked article is speculating that open-sourcing FoxPro might even create something of a renaissance in its usage -- at least in terms of the xBase language used within it.  I'm thinking that's a touch optimistic, but I'm also more than willing to be proven wrong.  If nothing else, it might at least pave the way for people to freely migrate away from FoxPro and towards a solution that is open-source by default, like MySQL.

My original coverage of Vista's backup utility was just a walk-through of its features, but as time has gone on it's become clear to me that many of the people who used the original NTBACKUP utility (or a third-party backup application) are really not happy with the new Vista backup tool.  They find it too simple (shilling for simpleminded), and want to have more control over exactly what gets backed up and where to.  In this episode of "What Happened To...?" I'll address these problems and suggest some workarounds.

1. What's the Vista Backup Utility and What's the Problem with It?

In an earlier article (linked above), I wrote about the Windows Vista Backup Utility, which is designed to back up documents and data to an external device (a hard drive, CD-ROM or network location), either on demand or on a schedule.

The single biggest problem that most people have with Vista Backup is the lack of flexibility in terms of what is backed up and from where.  In the original NTBACKUP tool, you could choose exactly what files and directories were backed up, and where to back them up to:

In Vista Backup, you can only choose one or more drives to back up from, and what TYPES of files from each you want to back up:

This creates some serious problems, since it's now very difficult if not outright impossible to include some specific file types and exclude others.  For instance, the "Compressed files" option includes .ZIP and .CAB files -- which I do want to back up -- but also includes .VHD (Virtual PC's Virtual Hard Disk format) files, which I don't want to bother with.  There's no way to include one but exclude the other, so I'm stuck backing up both of them unless I move the .VHD files to another drive (which I really don't want to have to do!).

So the biggest problem with the Vista backup tool is its relative inflexibility.  It's been made much easier to use, but at the cost of being that much harder to manage.

2. How Do I Even Know What's Been Backed Up?

Another complaint I've heard about the new backup tools is that it's too hard to tell if a given file has been backed up, as there doesn't seem to be a central manifest of backed-up files.

The best way to find out if a given file has been backed up is to right-click on the file, select Properties | Previous Versions, and wait for the File Versions list to populate.  There's no visual feedback that indicates if the list is still populating itself, so you may need to sit tight for a bit before everything appears.

 The "Location" column will tell you where previous versions of the file reside.  If you see a "Backup" entry for the last backup date (which is visible in the main window of the Backup and Restore Center, then that file's been backed up.  (If the "Location" column reads "Shadow copy" for a particular revision, that means that revision is stored in that volume's shadow copy repository.)

3. What's To Be Done?

There's a few things you can do in lieu of using the regular backup utility.

1. Use NTBACKUP once more

After Vista was released to the public, Daniel Petri did some research and found a way to get NTBACKUP running in Windows Vista.  It involves obtaining a copy of NTBACKUP and some support files from an existing copy of Windows XP, and then activating the Storage Manager service in Vista (which is not normally running by default).  If you're familiar with NTBACKUP, or already have backup files in the NTBACKUP format, you can use it more or less as you did in XP and Windows 2000.  You should not use it to make or restore System State backups, though, and (as with many legacy apps) NTBACKUP runs best in admin mode.

Many people also complained that people couldn't restore backups made with the classic NTBACKUP program.  To that end, Microsoft decided to release a free download for all those who had a licensed copy of Windows, which allows you to restore NTBACKUP repositories in Vista without needing NTBACKUP itself.

One major disadvantage of using NTBACKUP is that the files generated by NTBACKUP cannot be restored by the Vista recovery disk; you have to restore them by using NTBACKUP itself or with the above utility.  Also, you can't perform full system backup and restore with it as per the Vista backup utility.  Finally, NTBACKUP requires the Storage Manager service to be installed and enabled (there are instructions on how to do this within the download).

2. Use a third-party Vista-aware backup program.

If you don't want to use NTBACKUP, you can always turn to a third-party backup solution, as long as it's specifically marked as being Vista compatible.  Unless the app is Vista-aware, there will be a marked lack of integration with Vista (no backup notification, etc.), and backup sets may not be generated or restored correctly.

3. Change your storage habits

This is the least appealing option, and the most cumbersome.  If you have more than one drive, you can move everything that you don't want to have covered by the backup utility to another drive, and let Vista's backup automatically handle what's in your Users folder.  Again, though, this is probably not going to sit well with people unless they had planned on breaking down and remodeling their work habits in the first place.

After literally decades, Microsoft has finally set termination dates for one of its longest-running products -- no, not Windows, but FoxPro, the database-development product that survived due to a ferverent core of developers who continued to use it.  The product will continue to be supported through 2015 (!), with standard support ending in 2010.

I'm kind of amazed people were still developing for FoxPro -- not just because Microsoft has been trying to steer people away from it for years now, but because so much of what was possible in it has also been possible in the .NET framework, or through the SQL Server Desktop Engine.  I suspect a lot of the embedded applications for FoxPro (such as government offices) were a big part of the reason why they couldn't kill it off completely -- possibly for the same reason COBOL is still taught as an actual programming language and not simply confined to a museum exhibit somewhere.

The line "Brown Foxes Run Fast" was actually something that surfaced as an Easter egg on one of the Windows 95 beta-test CDs.  There was a point in 1995 when I was getting literally one or two of those a week; I wound up wallpapering an entire side of my office cube with them.  On one of the releases, I found a number of hidden audio tracks saying cryptic things like "This is a rather bland message!" and "Brown foxes run fast."  Many people believed the latter to be an allusion to FoxPro.  I couldn't think of a better theory.

In the middle of all the bad news about Windows Genuine Advantage, there are a great many stories about it working exactly as intended.  Ed Bott has one, wherein a user who got a bogus copy of XP was able to get back on his feet after WGA warned him that his Windows installation wasn't genuine.  (I should point out that the computer in question had a legit license key on the chassis; it's just that the copy of XP installed by the tech never used it.)

By the way, it doesn't surprise me that the rip-off in question was engineered by someone working for CompUSA, who are now shuttering half of their stores across the country.  (David Pogue had nothing nice to say about them, either, and neither did most of the people who responded to his article.)

In this installment of Whatever Happened To ... ?, I'll be talking about something that I touched on before in the past -- the way Windows Vista's defragmentation application has been simplified rather drastically.  There are some possible ameliorations, however.

[Note: this entry contains some previously published information in a new form.  If you've read the earlier piece I did on this subject, I encourage you to revisit this one.]

1. What's the Defragmentation App?

In Windows XP, the Defrag app was either the command-line or GUI application that allowed you to defragment the computer on demand.  It was also possible to set up the Task Scheduler to run the command-line version of Defrag on a regular basis.

2. What Happened to the Defragmentation App?

In Windows Vista, the Defrag GUI was replaced with something completely different:

You'll notice that one of the things now missing is any kind of statistics or progress bar.  This caused a great deal of annoyance among experts, who felt they were being deprived of valuable information.  What's the point of having a tool like this if we can't even see the progress meter?

Here is why this issue is not as obvious as it might seem:

1. Automatic scheduling

One of the things that has been added is an automatically scheduled defragmentation job that runs once a week.  In other words, a user who runs Vista and knows nothing about defragmentation will have their drive defragmented for them in the background during periods of low activity.  It means one less thing for everyone, expert and amateur alike to worry about.

2. Generic defragmentation statistics are not always useful

Second, the more I learn about fragmentation on a drive, the more I realize there is no one convenient metric that you can use to describe how badly a drive is fragmented.  You could talk about individual file fragments, but what if they're for files that are rarely accessed?  Or free space fragmentation, but what if the free space "fragments" are so large (50MB or more) that it's not going to create an issue for any new file written in it?

I guess it might be possible to come up with some kind of global "fragmentation index" -- like the way many of the newer anti-malware programs have some kind of "threat index" or "system protection index", but in the end I'm not sure that gives us any more useful information than we had before.  If a drive is fragmented badly enough that it's impacting performance, then we tend to take a fire-and-forget approach to the whole thing: we defrag the drive and move on to doing more serious work.

3. Defrag progress statistics can also be misleading

But what about the progress display?  Here's a list of the reasons for eliminating the display (and the lack of any progress data), straight from the horse's mouth:

The new interface seems “dumbed down.” Why remove all the detail?
...one of the biggest and consistent complaints we had from users (broad sample here from home users to experienced IT Pros) in the past was that a vast majority of them had no idea what the detailed fragmentation statistics they saw meant. The Windows XP graphical view also had some limitations and inaccuracies that prevented it from being included in Windows Vista. If you really want to keep a close eye on fragmentation, I’d recommend using the command-line tool Defrag.exe. [Emphasis mine]

Why was the defrag progress indicator removed?
Part of the problem with the Windows XP defrag tool was that percent complete was not accurate or meaningful. Depending on the phase of defrag, 1% of progress could take from several seconds to minutes, which made the progress indicator highly unreliable. The difficulty here is that since defrag is a multi-pass process (multiple iterations of file defragmentation and free space consolidation) there is no way to accurately predict when defrag will complete since the number of loop iterations and how long each takes are highly dependent on the layout of the files on the volume, the level of file and free space fragmentation, and the other system activity. While I agree that having no progress is bad, misleading progress I believe is worse. Also, the idea behind the new automated defrag is that users will not have to think about it not worry about the progress it is making. With defrag running regularly, the system will be close to optimal levels of fragmentation, and subsequent defrag runs should not take long. [Emphasis mine]

I should note that there has been a significant change to the way the defragmenter "sees" fragments on NTFS volumes.  Any given file fragment that is larger than 64MB is not consolidated by default -- it's left as-is.  This is because a fragment that large is probably something that's read sequentially anyway, so it probably won't appreciably improve the performance of the file if you forcibly collect all of its fragments and line them up.  (A file that large also doesn't tend to get read all the way through very often.)

3. What's To Be Done?

1. Leave it alone

This sounds like blunt, almost know-nothing-ish advice, I know.  But from what I have seen it's better to simply let Vista deal with fragmentation in its own way and at its own pace than to try and micromanage the whole thing (and with questionable performance returns).  In the long run, your time will be better spent on other things.

2. Use the CLI defrag

If you must defrag manually, use the CLI defrag command.  For a really aggressive cleanup, run it once a month with the following options, preferably before you go to bed:

defrag c: -v -w

This produces a verbose defragmentation report and forces a full consolidation of files.  It's actually useful to do this if you just upgraded from an earlier version of Windows and the drive is a mess of fractured directories, but in my opinion the full defrag is a waste of time if you do it more than once a month.

If you want to defragment with some kind of visual feedback, you can schedule the CLI version to run with a window [how?]

If you just want a fragmentation analysis, run defrag c: -a -v, which returns detailed statistics about the drive.  The results are not graphical, but in my opinion they are more useful than looking at a graphical map.  Again, a map can be misleading, since it may not give you a valid picture of what actual impact the fragmentation has on the drive.  The stats in the analysis includes things like the total available free space or the single largest contiguous free space on the drive, two things that are immensely important as far as fragmentation goes; the less free space you have, the more of a problem fragmentation is going to be and the harder it will be to defragment in the first place.

3. Use CONTIG to defragment individual files

If you want to defragment specific files, pick up the Contig utility from Sysinternals.  It's a little tool which defragments a single file through the Windows file-system APIs.  Contig is also used by the Power Defrgamenter utility, which can be used to defragment a single file, a whole directory, or a whole drive by using Contig.  It's most useful if you have a large file in several fragments that are not being defragged conventionally, but you don't want to go through the time and expense of running defrag with the -v switch to clean it up.  (Note that you'll probably need to run these programs as admin for them to work properly.)

4. Use a third-party, Vista-aware defragmentation app

I am putting this option last simply because I've grown less personally convinced that third-party defragmentation apps are crucial in a desktop environment.  The situation with servers is different, however; there's more than a few scenarios where a server can benefit from having some kind of fairly aggressive defragmentation.  (This is a topic that deserves its own discussion so I won't go into it here.)

Most of the commercial defrag programs have been fairly well-documented, like Diskeeper, but there are a few free ones that are worth mentioning -- JKDefrag, for instance, which does have a progress display and a great many other useful features besides.  One flaw: as of its current version it does not leave alone files tagged for prefetch, but the author is aware of this and may be including provisions for it in future releases.