Ravenous RAM use.

[Paul]

So it looks like chkdsk doesn't roll back its physical memory commitment
(into the pagefile or by dumping or reducing tables) to make room for
other processes. That sucks.

Did you perform only a file integrity scan or did you do a low-level
sector test (with the /r command-line switch or enabling the "Scan for
and attempt recovery of bad sectors" option)? Testing for reliable
sector read/write incurs more overhead. See:

http://support.microsoft.com/kb/2264358

File integrity checking is, well, of some use but when I run chkdsk it's
always with the /r command-line switch. Because that includes sector
testing (on both allocated and unallocated sectors), it takes a long
time to complete and seems foolish to have files opened since the file
you're working on could be on a sector found bad.

This was a read-only chkdsk. In the interests of time. No /r or anything.
It would have taken a while to do that over USB2.

First I tried chkdsk on C:, and that wasn't running long enough, which
is why I constructed a partition just for testing. And I figured a few
more files wouldn't hurt.

Paul

 

[Char Jackson]

File integrity checking is, well, of some use but when I run chkdsk it's
always with the /r command-line switch. Because that includes sector
testing (on both allocated and unallocated sectors), it takes a long
time to complete and seems foolish to have files opened since the file
you're working on could be on a sector found bad.

I wonder how much risk is there, though. If the file is open, it was
apparently read successfully at the time that it was opened. If you do
a chkdsk /r and find that one of that file's sectors is now bad, when
you save the file it should transparently use a mapped sector instead
of the bad one, or at least that's my understanding.

If you close the file without saving it, you may lose data, but it
seems that you would have lost that data anyway. After all, chkdsk
doesn't cause sectors to go bad, it only identifies those that have.

What's the scenario where having a file open during chkdsk is risky?

 

[Rob]

What I meant was that I have no problem with a program using all
available memory, since memory is there to be used, after all, but to
characterize a program that uses 12 GB of RAM as "efficient" is about
as out of whack as it gets. That's the opposite of efficient.
Additionally, suggesting that using 12 GB of RAM "will make it operate
much faster", well there's simply no evidence to support that claim.

That's why I figured he was only being sarcastic, and not to be taken
seriously.

Perhaps it should have been phrased "making extremely efficient use
of available resources". Any program which reads large amounts of
data from a hard drive should always operate much faster if it can
read large chunks into RAM in one go. Using less RAM would mean
more individual drive accesses which will slow the process down
considerably due to the fact that rotational latency has to be taken
into account as well as potentially more track to track head movement -
both slow, mechanically limited operations.
The best example of this would be a defragmentation program. If a
system had as much RAM as HD space (theoretically possible, but not
for the thin of wallet!), defragmentation could take seconds rather
than hours - 1 HD read operation (whole drive), defrag in RAM (several
order of magnitude faster than equivalent HD R/W operations), one HD
write operation.
HTH

 

[Paul]

Perhaps it should have been phrased "making extremely efficient use
of available resources". Any program which reads large amounts of
data from a hard drive should always operate much faster if it can
read large chunks into RAM in one go.

But the system already has this, in the form of the system file cache.
And it has access to as much RAM, as is currently free. And the system
file cache, given a chance, is virtually transparent to programs. You
get the caching action, even when you don't ask for it.

The difference is, if another program starts to run, the system file
cache lets go of the cached RAM instantly, invisibly to the user. The
system file cache would shrink to zero, if enough new programs
were asking for memory. It's a usage of memory, which doesn't
affect the user, and only serves to improve performance.

The chkdsk application, seems to be doing this caching privately.
They've implemented their own cache (or at least, memory consumption
is proportional to file system operations). And unlike the
system file cache, chkdsk keeps the "commit" portion even when the
computer is under memory pressure. And due to each new file operation
attempting to use their private implementation, chkdsk continues to apply
memory pressure to the system while it's running. So if I start my image
editor GIMP, GIMP is steadily "squeezed" or asked to reduce it's memory
usage, so chkdsk can go crazy. In Resource Monitor, I can see both programs
doing "page faults", as they're effectively thrashing each other's usage
of the computer.

I have no problem with a program attempting to implement its own cache,
as long as the implementation isn't too aggressive. And in this case,
chkdsk has "bad table manners". If four copies of chkdsk were sitting
at the same dinner table, they'd be stabbing one another with
dinner forks, to prevent their competitors from eating. Nobody
would get to eat, and there'd be a lot of fork wounds.

Paul

 

[Char Jackson]

Perhaps it should have been phrased "making extremely efficient use
of available resources". Any program which reads large amounts of
data from a hard drive should always operate much faster if it can
read large chunks into RAM in one go.

I agree with you in theory, but as I said above, we have no evidence
to support that position in this case. The key being, "in this case."
Ask any app developer if he or she can code an app that uses all
available resources while running slower than a similar app that uses
far less resources. It's dead simple to be that inefficient.

Using less RAM would mean
more individual drive accesses which will slow the process down
considerably due to the fact that rotational latency has to be taken
into account as well as potentially more track to track head movement -
both slow, mechanically limited operations.

We're talking about chkdsk, right? I see very little reason to do more
than read a track, compute checksums, write a track, read the same
track to be sure it was written correctly, etc. (Greatly simplified,
of course.) My point being, I see no real speed or performance
advantage to reading multiple tracks into memory and just leaving them
there. Once a track is verified it should be dumped and the memory
reused. Multithreading, processing multiple tracks in parallel, can
speed things up, but I still don't see the advantage of repeatedly
allocating more RAM to this particular task. It seems to be a bug, or
at least very poor coding.

The best example of this would be a defragmentation program. If a
system had as much RAM as HD space (theoretically possible, but not
for the thin of wallet!), defragmentation could take seconds rather
than hours - 1 HD read operation (whole drive), defrag in RAM (several
order of magnitude faster than equivalent HD R/W operations), one HD
write operation.
HTH

That's probably not a great example, in this case.

 

[VanguardLH]

I wonder how much risk is there, though. If the file is open, it was
apparently read successfully at the time that it was opened. If you do
a chkdsk /r and find that one of that file's sectors is now bad, when
you save the file it should transparently use a mapped sector instead
of the bad one, or at least that's my understanding.

If you close the file without saving it, you may lose data, but it
seems that you would have lost that data anyway. After all, chkdsk
doesn't cause sectors to go bad, it only identifies those that have.

What's the scenario where having a file open during chkdsk is risky?

Not using /r means you are checking the integrity of the file system,
NOT the hard disk and its sectoring and readability. If a problem is
found in the file system with an orphan and your file was opened from
there, and after chkdsk deallocates the orphan (file or folder) from the
file system by putting it into a filexxx.chk file, where's the file in
the file system that you still have open in memory in your viewer app?
What happens depends on any graceful recovery mechanism in the view app
on what it does with what it has in memory. Then there are the
cross-indexed defects in the MFT that chkdsk fixes but you might have
that file open when chkdsk disconnects it from the cross indexed
entries.

It could be chkdsk prevents this problem by refusing to start if it
cannot lock the volume. http://support.microsoft.com/kb/315265 says "If
one or more of the files on the hard disk are open ... Chkdsk cannot run
because the volume is in use by another process". It is because of
chkdsk altering the MFT that apps having opened files could result in
either losing that data or causing later corruption when the app tries
to save the file in memory but the MFT has been changed in the meantime.
Well, if chkdsk is going to lock the volume to protect the file system
which means you can't have files open then, one, it's the low-level
maintenance task that I mentioned and you should be doing it alone and,
two, the scenario of where chkdsk can alter the MFT while files are open
has been handled so my concern is moot.

 

[Rob]

But the system already has this, in the form of the system file cache.
And it has access to as much RAM, as is currently free. And the system
file cache, given a chance, is virtually transparent to programs. You
get the caching action, even when you don't ask for it.

The difference is, if another program starts to run, the system file
cache lets go of the cached RAM instantly, invisibly to the user. The
system file cache would shrink to zero, if enough new programs
were asking for memory. It's a usage of memory, which doesn't
affect the user, and only serves to improve performance.

The chkdsk application, seems to be doing this caching privately.
They've implemented their own cache (or at least, memory consumption
is proportional to file system operations). And unlike the
system file cache, chkdsk keeps the "commit" portion even when the
computer is under memory pressure. And due to each new file operation
attempting to use their private implementation, chkdsk continues to apply
memory pressure to the system while it's running. So if I start my image
editor GIMP, GIMP is steadily "squeezed" or asked to reduce it's memory
usage, so chkdsk can go crazy. In Resource Monitor, I can see both programs
doing "page faults", as they're effectively thrashing each other's usage
of the computer.

I have no problem with a program attempting to implement its own cache,
as long as the implementation isn't too aggressive. And in this case,
chkdsk has "bad table manners". If four copies of chkdsk were sitting
at the same dinner table, they'd be stabbing one another with
dinner forks, to prevent their competitors from eating. Nobody
would get to eat, and there'd be a lot of fork wounds.

Paul

I see your point Paul, but if a user is having to run chkdsk,
they are doing this for a reason, very likely a suspected
faulty drive or corrupted filing system.
It would be exceedingly dangerous for the user to be running
other applications at the same time (a little stupid, even), so in
this case I don't see a problem with chkdsk using all available
memory.
Best,

 

[Paul]

I see your point Paul, but if a user is having to run chkdsk,
they are doing this for a reason, very likely a suspected
faulty drive or corrupted filing system.
It would be exceedingly dangerous for the user to be running
other applications at the same time (a little stupid, even), so in
this case I don't see a problem with chkdsk using all available
memory.
Best,

Maybe it's my background, but let's try a few guiding principles.

1) As a designer, I don't know what users will do with the
system I design for them. I have to plan for every
eventuality.

2) As a corollary, I will use resources sparingly, so that
the design can "do more, with less".

I don't have a computer science degree, but I have written
a bunch of useless utilities, and those are the ideas
that guided my design. In fact, those ideas have guided
my designs, whether they were hardware or software.

I don't see why chkdsk should be any different. I've already
described use cases that ran afoul of this design. Namely,
the wish to chkdsk four separate physical drives in parallel
(to cut the checking time by a factor of four, parallelism).

I don't know all that goes into checking disks, but let's
take a stab at it anyway.

1) I want to cache the $MFT. I checked a disk here,
and that was ~600MB or so. I don't know if random access
on the $MFT is absolutely necessary, but that's just to give
an idea of a large structure I've heard about.

2) If I'm verifying each data cluster on the disk, I need
a small buffer during the read. All I really need, is the
CRC status from the disk drive, rather than the data itself.
I don't think chkdsk has any way of evaluating user data.
All chkdsk is interested in is file system integrity (all
links point to where they're suppose to).

So now my memory usage is bounded, by $MFT plus my small buffer.
It doesn't need to grow.

And for the design to be flexible, I have to make sure it can
run on a "small RAM machine". I have to consider the possibility
that only half of the $MFT can be cached (big MFT, small RAM).

So when a user reports 11GB of memory usage, I ask the question
"how wisely is that resource being used ?". And the answer is, not
wisely. Most of that RAM is being held captive, for no reason.
It's not being used (the designer knows the data in RAM, 10.4GB
of it, is now stale and won't be visited a second time).

Now, I'd let this go, *if* the chkdsk cache responded to
system memory pressure. In the same way that Windows 7
indexing has a "backoff" feature, where to ease user anguish,
the indexer process uses less CPU, when user activity is
detected. That was feedback from users, stating they wanted
"less indexer in their face".

The thing is, the previous chkdsk was a relatively good
design, in that it "didn't make a name for itself". It
didn't have outrageous resource usage. As a user, if
I want to run four copies on my WinXP machine, I wouldn't
think twice about it. With the chkdsk on Window 7,
mentally I think "red flag, engineering issue, caution"
when I think of a chkdsk run. Now that I've run the
experiments, I realize it's "start chkdsk, and walk away
from the computer". That's a piss poor usage model !!!
Especially if I'm checking that USB drive I just plugged in,
and I have other work to do. Imagine I'm in a computer
shop, and trying to work on more than one job at a time.

Paul

..

 

[Paul]

I am doing a chkdsk on my 1Tb HHD (not the system disk) and I see it's
using 12GB RAM..! The RAM is OCZ DDR3PC312800Inteli7
OCZ3X1600R2LV6GK. Is it normal for a chkdsk to use this much memory?
Peter

I've been working on your problem a bit, and I came up
with an unbelievably complicated solution

I tried a few things first. Like the "testlimit" program,
in an attempt to tie up the RAM and keep chkdsk from getting
it. But while I thought there was a parameter to set how much
RAM testlimit uses, there wasn't one. And in any case, chkdsk is
slightly too aggressive anyway, for that to work.

One of the offhand suggestions I read, was to use a 32 bit
version, or use an environment where it would be working in
a 32 bit space. On my x64 laptop, I saw no difference between using
the "GUI launched" chkdsk versus the "command prompt" way of running
chkdsk. They both used RAM in a significant way. The GUI launched
version, runs inside Explorer, while the command prompt way
runs separately. Made no difference for this problem.

Next, I read about a program called "corflags", which can
set flags on the header of .NET programs. But after getting
a copy of that, it turned out chkdsk isn't the right kind
of program for it.

To tell the files apart, I ended up using "file.exe" from the
GNU coreutils, which would report "PE32" for 32 bit executables
and "PE32+" for 64 bit ones. But the file.exe program,
couldn't report any other header flags.

Next stop, was to get an actual 32 bit version of chkdsk. To
do that, I downloaded a DVD from digitalriver, Win7 SP1 Home Premium x32
and had that do an install inside a virtual machine. I stopped the
install after the copy stage (just after the first reboot), then used
a Ubuntu LiveCD to locate and extract C:\Windows\System32\chkdsk.exe .
That's a 16000 byte file, compared to maybe 36000 bytes for the
64 bit version already on my laptop.

When I ran that, it still didn't stop hogging RAM. I did a bit
more reading, and discovered why. A 64 bit application, running
on a 64 bit OS, can use all the memory. A 32 bit application
running on a 64 bit OS, can use 2GB if LargeAddressAware is not
set, and 4GB if it is set. Since my laptop has 3GB of physical RAM,
I seemed to be seeing that the copy of chkdsk.exe I got from
the 32 bit half-install, had the LargeAddressAware flag set.

That meant locating a copy of "editbin", a program that can
edit executable files and change the header. It's similar in
some ways to "corflags", but is for regular executables.

editbin /largeaddressaware:No chkdsk.exe

What that does, is change the 32 bit application from being
able to grab 4GB, down to 2GB. I didn't do that to the regular
chkdsk on my laptop, but to the copy I "borrowed" from the
32 bit half-install.

I then tested the applications a bit, using my ~110GB partition
containing 3.2 million test files. Now, my machine has 3GB of
RAM, and maybe around 2400MB is as far as an application can
go, without stomping on OS space. I repeated the first run,
to see what impact that has, and the 64 bit version of chkdsk
was able to grab a bit more memory on the second run. So if
reporting results, it pays to "warm up" the OS by doing one
run before gathering statistics.

x64 (greedy gus) runtime 9:04 max_ram 2004MB

x64 (greedy gus) runtime 9:08 max_ram 2276MB <--- used a bit more

Next, I tried the 32 bit version of chkdsk.exe, the one
where I'd used editbin to change the LargeAddressAware flag
in the header.

x32s (slim pickens) runtime 9:07 max_ram 1772MB

On my WinXP x32 machine, I typically see 1800MB as the
practical limit for things like really old Photoshop.
So the 1772 value seems reasonable for the "2GB" type limit.

I also ran the unmodified x32 version, but this test
run was ruined by some background activity. So the
runtime is wrong. The max_ram is still indicative,
that the program would have liked to chew into more RAM.
If I'd had an 8GB machine, it probably would have
grabbed around 3400MB or so. It stopped at 2316MB
because there was no more free memory to be had.

x32 (needs diet) runtime 11:09 max_ram 2316MB

So for the "least RAM hungry" solution, getting a
32 bit version of chkdsk from a 32 bit install, plus
using editbin to turn off LargeAddressAware, does
the best job possible.

The other thing I found comical, was the 32 bit and 64 bit
applications, didn't give the same results! Earlier today,
when I did a run with the 32 bit one, it found a couple
files with an "index problem". Running the 64 bit version
immediately afterwards, shows no such problem. The two
"damaged" files appeared to be associated with the AV software.
So it probably wasn't a real difference, but some interaction
between what the AV was doing, versus what chkdsk was doing.
The runs I report above, had no repeat of that (all had no
errors found), so the problem "disappeared" on its own.

HTH,
Paul