Defragging System Volume Information

[SC Tom]

The act of moving the SVI folder results in losing the System Restore
points. So instead of defragging the SVI folder, just turn off SR, lose
the restore points, and reenable SR. Since you'll lose the restore
points if you move this folder, why bother moving it? Just delete
what's inside to recover the disk space.

Not sure if that's just an effect of defrag on a 64-bit OS or not, but I used Auslogics on WinXP and Win7 (both 32-bit)
to test that, and am happy to report that the SVI was defragged (showing no fragmented files) and that all my SR points
are still showing, and the ones from two days ago still worked. Now, whether or not Auslogics moves the folder or not, I
don't know, but it did move the fragments in (of?) the folder to make the files contiguous. The only files that show up
as fragmented on the entire drive after completion are a couple of files in my temp folder, and one of them is the
defrag log file.

 

[VanguardLH]

I remember running into that situation in the late 90's, but then
Partition Magic and its cousins came along, completely eliminating the
defrag as a separate step in reducing a partition's size. Those were
the days! For the past 10 years or so, most partition managers
silently take care of that for you.

I have seen info about how to disable the Journal feature of NTFS but
running "fsutil usn deletejournal /d C:" so I'm sure a defragger or
partition manager could do this, too. Apparently any app (with admin
privileges) can enable or disable the NTFS journal (as noted at
http://www.microsoft.com/msj/0999/journal/journal.aspx by using the
DeviceIoControl function). That would eliminate the otherwise unmovable
$LOGFILE files. I don't see that a reboot (restart of the OS) is
required after deleting/disabled the USN journal
(http://technet.microsoft.com/en-us/library/cc788042(WS.10).aspx) but it
does mean that you're tossing some state info afforded by NTFS. "fsutil
usn queryjournal c:" will tell you some info but I doubt it's of any
value to you and it doesn't specify if the $LOGFILE is contiguous or
sliced across the partition.

My guess is that those running defraggers and partition managers happen
to be logged under an admin-level account so those apps could disable
journaling. But can you guarantee that all defraggers and partition
managers properly nullify features in NTFS so not to cause corruption?
I have seen users complain about boot-time defraggers and partition
managers that left them with a Stop 7B (inaccessible device) due to NTFS
corruption that left their host unusable because they cannot boot from
their hard disk.

Also, as Paul points out, resizing a partition has that utility moving
files at the end of partition to somewhere below the proposed end of the
new partition size. That's not the same as defragging a partition. Do
you know that defraggers and partition managers disable the NTFS journal
so their boot-time operation won't have any exclusively locked system
files to contend with? I've seen several users of various defraggers
complain that $UsnJrnl is huge and won't defrag. If the defragger could
simply delete it then why didn't the defragger do it? If there were no
consequence of deleting the change journal, why didn't the defragger
just go ahead and do it rather than make the user run the fsutil program
to do it manually?

From what I've seen so far, safe defraggers use the ioctrl call in
Windows but that won't let you defrag the usn journal because it is
locked for exclusive access. I've also seen it claimed that deleting
the usn journal causes no problems. It could have records of changes
made months ago on old files and is really a recovery mechanism (but
then I'm not an expert on NTFS). The file can get huge and waste a lot
of disc space (I think there is a registry setting on its max size) and
this is a rollover logfile: old entries get pushed out as new ones get
pushed in. However, I've seen it mentioned with use of the replication
service which could affect operation of other services (e.g., servers
hosting the fault-tolerant DFS for root or child node replicas); see
http://www.techrepublic.com/article/the-ins-and-outs-of-the-windows-file-replication-service/5053993.
So there appears some problems with just blowing away the usn journal
(but may be a concern mostly just for server versions of Windows).

Although you can delete the usn journal using fsutil, it appears that it
just momentary. It is an auto-created file so more changes means the
file gets created to record those changes. Maybe that's why you must do
a boot-time defrag to delete the journal along with keeping the OS
somewhat quiescent to prevent generating a journal file (or limit its
initial occupation to the first few unused clusters in the file system
as it begins to grow).

By the way, I have used partition managers that were updated in the last
decade and still ran into errors during their resize operation. Alas,
they provide so little info on why they failed to perform the resize
that the user doesn't know how to resolve the problem. Also, despite
the "silent" (requiring reboot) pseudo-defrag (move files away from end
of partition to get them under the boundary for the new end of
partition), users have ended up with unusable partitions due to NTFS
corruption. Just because a partition manager makes it look easy doesn't
mean resizing isn't without its gotchas. The user easily makes their
choices, the program does it thing (perhaps requiring a reboot), and
then there's a failure which sometimes is not recoverable or causes lots
of problems thereafter. If you're lucky when the partition manager
fails to complete the resize, nothing got changed and you're still in
business or you lost little and what you lost wasn't critical.
http://www.google.com/search?q=+partition++resize+error+fail. It
happens and with partition managers created/updated in the last decade.

While I mentioned doing my own defrag before a partition resize (since
that'll tell me if there might be problems due to unmovable files), I
failed to mention that I also save an image backup unless there's
nothing critical in the partition(s) or it's of little value or it can
be recreated. Resizing isn't the ever-safe operation you make it out to
be just because the utilities make it easy to make changes. Easy to
select doesn't make the operation safe or provide complete or even
partitial recovery after a failure. I consider partition resizing a
hazardous operation.

 

[VanguardLH]

That surprises me. I don't see why defragging (which really isn't
necessary in the first place) would have any effect on incremental or
differential backups.

I'm not talking about logical file backups (where the file is read as a
long string through the file system, a new backup file is created with
its content, and the file gets restored through the file system which is
very similar to you running the 'copy' command in a command shell).
That's like what you get with the NT backup or Xcopy programs. It's
probably been 12-15 years since I did the old logical file backup method
ever since I went to paid/free imaging backup solutions. Since I
haven't done it that way for so long, it's not the default scheme I
think about. A defrag doesn't change the content of the file (other
than perhaps what's in its slack space but which is not normally
accessible via typical file I/O calls) so a backup method that merely
uses normal file I/O calls to read the file, store it, and then restore
it will result in the same file content wherever its clusters happen to
be inside the partition.

Sorry, but I failed to mention that I was talking about *image* backups
(and also not physical sector-by-sector exact image backups). Due to
the physical relocation of files caused by a defrag, all those moved
files become eligible for inclusion in an incremental or differential
*image* backup. Defragmentation changes the file locations on the disk,
and image backups reflect those changes. Say you do a full image backup
one day and the next day you do an incremental image backup but in
between you ran a defrag. Your incremental image defrag will be far
larger than expected (as you remember only moving the files and not
modifying their content) and could be nearly as large as your full image
backup.

If you read the FAQ or support pages for image backup programs, they'll
[should] tell you that a defrag will affect incremental and differential
backup size.

 

[Char Jackson]

I have seen info about how to disable the Journal feature of NTFS but

My guess is that those running defraggers and partition managers happen
to be logged under an admin-level account so those apps could disable
journaling. But can you guarantee that all defraggers and partition
managers properly nullify features in NTFS so not to cause corruption?

I wasn't claiming to guarantee anything. I was simply saying in my
experience the two-step "defrag then resize" operation was obsolete. I
haven't run into a partition manager that required it since the late
1990's. Then again, most of my experience in the last 20 years has
been with Partition Magic until about 2002/2003 (?) and with Acronis
Disk Director since then, but there have been a few others used along
the way when I didn't have access to my primary tools. None required
me to use a separate tool to move data before allowing me to resize a
partition.

I have seen users complain about boot-time defraggers and partition
managers that left them with a Stop 7B (inaccessible device) due to NTFS
corruption that left their host unusable because they cannot boot from
their hard disk.

Users complain about a lot of things that won't happen to most people.
The Internet is a big place. We don't know the details of those
anecdotes.

Also, as Paul points out, resizing a partition has that utility moving
files at the end of partition to somewhere below the proposed end of the
new partition size.

I thought it was me who pointed that out, but maybe I just silently
agreed and moved on.

That's not the same as defragging a partition.

Actually, it has very little to do with defragging, and maybe nothing
at all. It's simply a process of moving data from one place to
another. No attempt needs to be made to defrag anything.

Do
you know that defraggers and partition managers disable the NTFS journal
so their boot-time operation won't have any exclusively locked system
files to contend with?

Don't know, don't care, never had a problem and don't expect to. Any
decent partition manager will know how to take care of data that would
otherwise fall outside of the newly proposed partition boundaries. If
it can't do that, dump it ASAP and get another (better) tool. There
are many to choose from.

Resizing isn't the ever-safe operation you make it out to
be just because the utilities make it easy to make changes. Easy to
select doesn't make the operation safe or provide complete or even
partitial recovery after a failure. I consider partition resizing a
hazardous operation.

I think you're putting words in my mouth.

 

[VanguardLH]

I wasn't claiming to guarantee anything. I was simply saying in my
experience the two-step "defrag then resize" operation was obsolete. I
haven't run into a partition manager that required it since the late
1990's. Then again, most of my experience in the last 20 years has
been with Partition Magic until about 2002/2003 (?) and with Acronis
Disk Director since then, but there have been a few others used along
the way when I didn't have access to my primary tools. None required
me to use a separate tool to move data before allowing me to resize a
partition.


Users complain about a lot of things that won't happen to most people.
The Internet is a big place. We don't know the details of those
anecdotes.

An opinion from one user regarding their personal use has some but
little value when making blanket statements. Perhaps you haven't needed
to fix computers for other users.

Actually, it has very little to do with defragging, and maybe nothing
at all. It's simply a process of moving data from one place to
another. No attempt needs to be made to defrag anything.

But moving files with a partition manager that are exclusively locked
still runs into the same problems with defraggers moving those same
files. So basically the issues with defraggers are the same as for
partition managers when moving around the files to different clusters.

Don't know, don't care, never had a problem and don't expect to.

Yes, got that, you haven't run into the problem with a non-bootable
partition after using a boot-time defragger or a partition manager that
reboots and then proceeds to move about the system files along with
those within the NTFS file system. By the way, you (and I) really
didn't mention if you/me use NTFS. I use it exclusively except for USB
flash drives. You haven't run into a non-bootable partition after a
resize, I have, so the voting on personal experience is even; however,
other users do encounter problems so to dismiss them simply because you
haven't encountered them doesn't make them non-existent or impossible
and not even improbable.

Any decent partition manager will know how to take care of data that
would otherwise fall outside of the newly proposed partition
boundaries. If it can't do that, dump it ASAP and get another
(better) tool. There are many to choose from.

If defraggers cannot move around the unmovable files, just how is any
partition manager going to do it? Some defraggers can do it (at boot
time) but some users HAVE encountered problems thereafter. I'm not
saying it's a prevalent problem but it does happen. When it happens to
you (or you're stuck having to repair someone else's computer where it
happens), it suddenly isn't such rare a problem.

It hasn't happened to you. Good for you. It has happened to me. When
you get burned, you figure out how to prevent it, not just wait to see
if it happens again. Pain is an excellent learning motivator.

 

[Paul]

Any decent partition manager will know how to take care of data that would
otherwise fall outside of the newly proposed partition boundaries.

That was my point, of what I was doing with the Windows 7 built-in "shrink"
function. Unlike the third party competition, Microsoft didn't bother
moving everything that could possibly be moved (which is why is stops at 51%).
I found a defragmenter with the required capability and file movement policies
(ability to move metadata files to the left a bit). And by using multiple passes
of the two tools (defragmenter + Microsoft "shrink" function), I got the job done.

I presume a purpose built partition manager tool, would have to do
something similar, have the ability to move metadata around. Otherwise
it would get stuck like the Microsoft implementation.

In the picture here, I think it was the two block "grey thing" at around the 50% mark,
that causes the problem. I didn't keep any screenshots of what mine was doing.
The defragmenter won't push it all the way to the left either, so they're
following some rule about its position.

http://img841.imageshack.us/img841/7489/625201111431pm.png

*******

I found a utility which might have some fun value. It was mentioned
in the Wikipedia article on NTFS. (This is for if you don't own a
commercial defragmenter, and want to know where things are. It also
demonstrates fragmentation, in that fragmented files have more than
one group of sectors listed.)

http://support.microsoft.com/kb/253066/

http://download.microsoft.com/download/win2000srv/utility/3.0/nt45/en-us/oem3sr2s.zip

It's got a copy of nti.exe in it. If I run that on a NTFS (data) partition here
on my WinXP machine, it gave about a 4MB dump, similar to this. The
"two grey blocks" in the middle, could be $MftMirr.

File 0
Master File Table ($Mft)
$STANDARD_INFORMATION (resident)
$FILE_NAME (resident)
$DATA (nonresident)
logical sectors 32-25087 (0x20-0x61ff)
$BITMAP (nonresident)
logical sectors 16-23 (0x10-0x17)

File 1
Master File Table Mirror ($MftMirr)
$STANDARD_INFORMATION (resident)
$FILE_NAME (resident)
$DATA (nonresident)
logical sectors 428670424-428670431 (0x198cfdd8-0x198cfddf)

....

File 12441
\27a368f93559180042cd5c7058f99841\Setup\custsat_x86.dll
$STANDARD_INFORMATION (resident)
$FILE_NAME (resident)
$FILE_NAME (resident)
$DATA (nonresident)
logical sectors 421623008-421623087 (0x192174e0-0x1921752f)

So if nothing else, you can use it as a file listing tool

The volume I tested that on, is 857340855 sectors, and it's only
about half full. One peculiarity, is not every "File" is
listed, and the file numbering skips at one point. The
volume has 11097 files and 1311 directories or 12408 objects.
So the count is roughly in the right ballpark. Still, pretty
damn good for a program which is only 21,744 bytes in size.
Good things come in small packages

Paul