Ravenous RAM use.

[Peter Jason]

I am doing a chkdsk on my 1Tb HHD (not the system disk) and I see it's
using 12GB RAM..! The RAM is OCZ DDR3PC312800Inteli7
OCZ3X1600R2LV6GK. Is it normal for a chkdsk to use this much memory?
Peter

 

[VanguardLH]

I am doing a chkdsk on my 1Tb HHD (not the system disk) and I see it's
using 12GB RAM..! The RAM is OCZ DDR3PC312800Inteli7
OCZ3X1600R2LV6GK. Is it normal for a chkdsk to use this much memory?
Peter

You have 12GB of RAM for any process to consume?

 

[soup]

I am doing a chkdsk on my 1Tb HHD (not the system disk) and I see it's
using 12GB RAM..! The RAM is OCZ DDR3PC312800Inteli7
OCZ3X1600R2LV6GK. Is it normal for a chkdsk to use this much memory?
Peter

12 GIGS of RAM? Methinks you have misstyped something.
Although that amount is possible it is unusual what sort of kit you
running? Opinion is divided as to whether this RAM utilisation is by
design or a bug. Unless you are trying to run other things or at the
end of chkdsk it is not releasing this memory what is the problem?

 

[Paul]

I am doing a chkdsk on my 1Tb HHD (not the system disk) and I see it's
using 12GB RAM..! The RAM is OCZ DDR3PC312800Inteli7
OCZ3X1600R2LV6GK. Is it normal for a chkdsk to use this much memory?

Peter

The customer view. At least a few customers aren't happy,
that checking non-C: partitions, chews up all the RAM.

http://social.technet.microsoft.com...l/thread/a3cd974b-4758-44bb-b1c0-289ef813b2e6

The Microsoft view. Say "it's by design", and then blow
smoke up your ass.

http://blogs.msdn.com/b/e7/archive/2009/08/10/what-we-do-with-a-bug-report.aspx

I can't even understand, how holding gobs and gobs of disk structure
is an advantage. If we looked at the size of $MFT for example,
would we need 12GB to hold it ? Or, some smaller amount ?
In a quick check, I see around 275MB for $MFT on my laptop
C: drive (determined with the nfi.exe log I created some time
ago).

The operating system, already has a file cache, which works
transparently, and evicts cached structures whenever the
system is under memory pressure. If you start a program and
it needs RAM, the system simply updates the list of things
it thinks are cached in memory. And doing so, is virtually
instantaneous, and the user doesn't even know a cache is
present. Now, if chkdsk used the regular system file cache,
as structures were being read, there would be no design
problem at all. If instead, they do private allocations,
we'd get the symptoms people are seeing. So the question
would be, "why isn't the regular file cache, good enough?".

If this is "design intent", I don't see the "good design principles"
oozing out of it :-( If the design is really that good,
someone at Microsoft should be able to explain it.
Instead of describing how many engineers they fly around
on airplanes.

And to think that blog entry was written by Steven Sinofsky.

http://en.wikipedia.org/wiki/Steven_Sinofsky

I'd like to hear an opinion from Mark Russinovich. I wonder
what he thinks about this "feature".

Paul

 

[Paul]

There is a workaround mentions on dailytech.

"No point delaying. The issue is only on the 64-bit version of Windows 7
and it's not a true memory leak.

The version of chkdsk.exe is the same on both 32-bit and 64-bit installations
of Windows 7, but doesn't seem to be compiled for 64-bit use or being run
through the 32-bit emulator properly. So, when memory is addressed it goes
into a loop and never stops incrementally allocating a memory block. The issue
doesn't happen when using the 32-bit version of command prompt
("C:\Windows\SysWow64\CMD.exe").
"

I think what that poster is suggesting, is that by using a 32 bit environment
for the executable, you can stop it from using more than 2GB of virtual address
space. That's how I interpret that idea as a "win". I don't think it is actually
fixing anything, merely placing a "32 bit road block" in the way.

When this question was asked previously, there was evidence that if
the memory was blocked artificially, chkdsk would stop chewing up the
RAM. But it's unclear to me, whether that is true or not. And nobody
tested my suggestion. You can use the Sysinternals "testlimit" or "testlimit64"
programs, to do your own RAM grabbing, to prevent chkdsk from getting it all.
What is unclear, is when you kill off the testlimit program, whether chkdsk
begins to chew on the freed up RAM. If it was a memory leak, it should do so.

The behavior is a bit puzzling. If it was a leak, I presume swap would
also get chewed up, until the system crashed. And while there are
reports of BSODs happening, that's relatively infrequent. Some people
with 8GB, claim CHKDSK will use 7GB. Which sounds like the program has
measured available physical RAM and stopped there. If the response is
measured, then "it's by design". It implies they designed it that way.

But the behavior is still not acceptable. For example, one poster described
running four copies of CHKDSK at the same time (one running per disk, as such
a process is disk limited and not CPU limited), and the programs would
fight for memory, leaving some of the instances starved for memory,
compared to their peers. And apparently, that was slowing down
all the instances. I'm still not convinced there is a "good design"
hiding in there.

If you do any further testing, and figure out a solution, please post back.
I'm not testing this one... I don't have big enough disks for this, or
good enough setups. (3GB RAM, 320GB disk)

Paul

 

[Peter Jason]

You have 12GB of RAM for any process to consume?

Yes; the motherboard (GA-X58A-UD7-2) had all the RAM slots, so I
filled them up. The only other application using lots of RAM is the
Adobe Premiere Pro which used 40% when finalizing a movie.

I am test the HDDs because two of them disappeared completely from
Windows Explorer and Computer Management. One subsequently
re-appeared on a reboot so I did a chkdsk on it. (no problems found).
The other HDD is still hiding. The rest is mystery. Still, the
missing disk(s) are active mechanically; I can feel them buzzing
and burping.

 

[Peter Jason]

There is a workaround mentions on dailytech.

"No point delaying. The issue is only on the 64-bit version of Windows 7
and it's not a true memory leak.

The version of chkdsk.exe is the same on both 32-bit and 64-bit installations
of Windows 7, but doesn't seem to be compiled for 64-bit use or being run
through the 32-bit emulator properly. So, when memory is addressed it goes
into a loop and never stops incrementally allocating a memory block. The issue
doesn't happen when using the 32-bit version of command prompt
("C:\Windows\SysWow64\CMD.exe").
"

I think what that poster is suggesting, is that by using a 32 bit environment
for the executable, you can stop it from using more than 2GB of virtual address
space. That's how I interpret that idea as a "win". I don't think it is actually
fixing anything, merely placing a "32 bit road block" in the way.

When this question was asked previously, there was evidence that if
the memory was blocked artificially, chkdsk would stop chewing up the
RAM. But it's unclear to me, whether that is true or not. And nobody
tested my suggestion. You can use the Sysinternals "testlimit" or "testlimit64"
programs, to do your own RAM grabbing, to prevent chkdsk from getting it all.
What is unclear, is when you kill off the testlimit program, whether chkdsk
begins to chew on the freed up RAM. If it was a memory leak, it should do so.

The behavior is a bit puzzling. If it was a leak, I presume swap would
also get chewed up, until the system crashed. And while there are
reports of BSODs happening, that's relatively infrequent. Some people
with 8GB, claim CHKDSK will use 7GB. Which sounds like the program has
measured available physical RAM and stopped there. If the response is
measured, then "it's by design". It implies they designed it that way.

But the behavior is still not acceptable. For example, one poster described
running four copies of CHKDSK at the same time (one running per disk, as such
a process is disk limited and not CPU limited), and the programs would
fight for memory, leaving some of the instances starved for memory,
compared to their peers. And apparently, that was slowing down
all the instances. I'm still not convinced there is a "good design"
hiding in there.

If you do any further testing, and figure out a solution, please post back.
I'm not testing this one... I don't have big enough disks for this, or
good enough setups. (3GB RAM, 320GB disk)

Paul

Mine is a 64bit system. I'll chkdsk another disk later today.

 

[Ed Cryer]

Yes; the motherboard (GA-X58A-UD7-2) had all the RAM slots, so I
filled them up. The only other application using lots of RAM is the
Adobe Premiere Pro which used 40% when finalizing a movie.

I am test the HDDs because two of them disappeared completely from
Windows Explorer and Computer Management. One subsequently
re-appeared on a reboot so I did a chkdsk on it. (no problems found).
The other HDD is still hiding. The rest is mystery. Still, the
missing disk(s) are active mechanically; I can feel them buzzing
and burping.

Does Windows recognise it to any degree at all?
Does it make the ding-dong sound when you plug it in? Any sign in the
lower pane of Disk Management? Anything under USB in Device Management?

If you can rule all those out, then it will look very much like a
hardware thing; dodgy cable connection, or failed innards of the drive
itself.

Trying it on another computer will be a great help.

Ed

 

[Char Jackson]

Yes; the motherboard (GA-X58A-UD7-2) had all the RAM slots, so I
filled them up.

I guess you're lucky they didn't sell you a server motherboard that
can handle 64 GB of RAM because you probably would have filled that
up, too.

 

[Dominique]

Peter Jason <[email protected]> écrivait

I am test the HDDs because two of them disappeared completely from
Windows Explorer and Computer Management. One subsequently
re-appeared on a reboot so I did a chkdsk on it. (no problems found).
The other HDD is still hiding. The rest is mystery. Still, the
missing disk(s) are active mechanically; I can feel them buzzing
and burping.

I've had a similar issue; my computer had a side panel removed and had
hard drives power wires outside the case, when my foot touched the outside
wires, 2 of the HDs would disappear (fortunately, not the system drive);
it's a 4 HDs system. I've put back the outside wires inside the case and
no issue since then (the side panel is still ... off).

So, in your case, it could be wiring or power supply issues?...

 

[Peter Jason]

Mine is a 64bit system. I'll chkdsk another disk later today.

I chkdsked another 1TB disk (about 1/3 full) and I got the same
behavior. From the "Resource Monitor" the "Used Physical Memory"
level was constant at 95%. The chkdsk took about 2 hours. No
problems found.

 

[soup]

the side panel is still ... off

Could be dangerous, air flow for cooling is usually regarded as having a
fully(ish) enclosed system. Having the side of may well mean that air
isn't flowing over certain components so they may well overheat.
There is also the interference issue,. And the issue of being able to
access the parts whilst they are energised.

c.f. Blanking plates on the case to cover slots in the case which
additional cards can use.

 

[Oldish Git]

I am doing a chkdsk on my 1Tb HHD (not the system disk) and I see it's
using 12GB RAM..! The RAM is OCZ DDR3PC312800Inteli7
OCZ3X1600R2LV6GK. Is it normal for a chkdsk to use this much memory?
Peter

Excellent news. This means chkdsk is being extremely efficient and using as
much
RAM as possible, which will make it operate much faster.
HTH

 

[Gene E. Bloch]

Excellent news. This means chkdsk is being extremely efficient and using as
much
RAM as possible, which will make it operate much faster.
HTH

About time someone mentioned that! Thanks, Oldish Git.

 

[Char Jackson]

About time someone mentioned that! Thanks, Oldish Git.

I assumed he was being sarcastic, no?

 

[VanguardLH]

Yes; the motherboard (GA-X58A-UD7-2) had all the RAM slots, so I
filled them up. The only other application using lots of RAM is the
Adobe Premiere Pro which used 40% when finalizing a movie.

I am test the HDDs because two of them disappeared completely from
Windows Explorer and Computer Management. One subsequently
re-appeared on a reboot so I did a chkdsk on it. (no problems found).
The other HDD is still hiding. The rest is mystery. Still, the
missing disk(s) are active mechanically; I can feel them buzzing
and burping.

http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778(v=vs.85).aspx

It is normal for a user process to use as much memory as the OS will
permit. While you mention it using a lot of memory, you never mentioned
if it was pegging the CPU at 100% to make unresponsive the computer.
Why wouldn't you want a process to have as much memory as it can instead
of choking it, making it use the pagefile, and slowing it down? You
wanted it to be throttled down to 2GB to save the rest for what? What
else needed the memory? Were these other programs that you wanted to
run concurrently with chkdsk refusing to load?

This is similar to when users wonder if they need more system RAM but
fail to mention if what they already have has already been consumed by
background processes or they need more multiple processes running
concurrently are are actually running out of system memory. If you have
RAM that is unused then it is wasted RAM. Why wouldn't you want a
process to suck up huge amounts of RAM if it benefits that program as
long as it doesn't starve your other wanted processes from also having
sufficient RAM?

We don't under what scenario you are running chkdsk. If you run chkdsk
but it requires a reboot to unlock the system partition then nothing
else than Windows, drivers, services, and chkdsk need the RAM so why not
let it use as much as it wants? If you are running chkdsk (without the
/f or /r switches) when inside of Windows, does it peg the CPU at 100%?
If so, at what priority is the chkdsk process running? Does running
chkdsk preclude you from concurrently running other programs?

What else did you need the memory for at the time you chose to run
chkdsk? When doing low-level maintenance, why are you trying to do
anything else on the computer? If it found a file to fix, it could be
the one you currently had open in some app which means you could lose
some or all of the file.

 

[Gene E. Bloch]

On Tue, 06 Dec 2011 11:15:07 -0800, Gene E. Bloch

I assumed he was being sarcastic, no?

Maybe sarcastic, but pertinent.

 

[Paul]

http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778(v=vs.85).aspx

It is normal for a user process to use as much memory as the OS will
permit. While you mention it using a lot of memory, you never mentioned
if it was pegging the CPU at 100% to make unresponsive the computer.
Why wouldn't you want a process to have as much memory as it can instead
of choking it, making it use the pagefile, and slowing it down? You
wanted it to be throttled down to 2GB to save the rest for what? What
else needed the memory? Were these other programs that you wanted to
run concurrently with chkdsk refusing to load?

This is similar to when users wonder if they need more system RAM but
fail to mention if what they already have has already been consumed by
background processes or they need more multiple processes running
concurrently are are actually running out of system memory. If you have
RAM that is unused then it is wasted RAM. Why wouldn't you want a
process to suck up huge amounts of RAM if it benefits that program as
long as it doesn't starve your other wanted processes from also having
sufficient RAM?

We don't under what scenario you are running chkdsk. If you run chkdsk
but it requires a reboot to unlock the system partition then nothing
else than Windows, drivers, services, and chkdsk need the RAM so why not
let it use as much as it wants? If you are running chkdsk (without the
/f or /r switches) when inside of Windows, does it peg the CPU at 100%?
If so, at what priority is the chkdsk process running? Does running
chkdsk preclude you from concurrently running other programs?

What else did you need the memory for at the time you chose to run
chkdsk? When doing low-level maintenance, why are you trying to do
anything else on the computer? If it found a file to fix, it could be
the one you currently had open in some app which means you could lose
some or all of the file.

I've just finished some testing here.

Setup:

Win7 Home Premium x64 OEM
Laptop with 320GB internal drive (C
3GB physical memory (two SODIMM)
External USB drive 110GB partition 3.2 million files (my testing partition)

I ran chkdsk by doing the old Properties:Tools tab thing and
selecting disk checking from there.

Using Resource Monitor, memory usage by chkdsk (inside Explorer) was
up around 2.7 or 2.8GB. It grew steadily until it ran out of
memory to grab (like it stopped at the remaining physical limit).
Page faults were close to zero, job finished about as quickly as
you might expect.

For this test, all other activity on the machine was quiet. Just
me watching Resource Monitor while the chkdsk run progressed.

*******

Test case #2

For this test, I simulate a user trying to work while chkdsk is
running. I open GIMP and create a 40000x3000 pixel blank image
using New from the menu. And start working within it. The image
is supposed to be around 1GB or so. Memory resident, is usually
less than that.

Well, now the gloves come off. Chkdsk doesn't relinquish any
memory (the "commit" stays at the top value, resident memory
drops, and fills again), and instead, it is making fresh requests
as the chkdsk run proceeds. I can't really work in GIMP, because
there are as many page faults against GIMP image editor, as
against chkdsk.

The long and the short of it, in this case, "nobody wins".
I can't actually work productively in the image editor, and
chkdsk isn't running quite as fast as before.

For most of the second test case, the disk light on the laptop
was on solid. That's due to the paging going on (thrashing between
the two programs). I can imagine running multiple copies of
chkdsk, to check separate physical drives, would end up with
similar behavior.

So the only test case where I got satisfactory results, was
running one copy of CHKDSK and walking away from the computer.
Since I was checking the external drive, it wasn't like I was
waiting on it to complete (C: was not involved).
I could have been editing an image from the C: drive, using GIMP,
but performance was so slow (lots of "not responding"), the machine
really couldn't be used for anything else.

*******

Now, if I do these things in WinXP, the machine behaves normally,
and I can actually multitask. Since the disk with the 3.2 million
files was prepped on the WinXP machine, I did actually try it out
over there as well. I could work normally, while the chkdsk was
working on the 3.2 million files.

As for where the 3.2 million files came from, I wrote a small
program to write out a set of files (took about five hours runtime).
I wanted lots of files, so the chkdsk run would last long enough
to observe memory usage.

*******

The Windows 7 design is a step backwards from the WinXP version of chkdsk.

Paul

 

[Char Jackson]

Maybe sarcastic, but pertinent.

What I meant was that I have no problem with a program using all
available memory, since memory is there to be used, after all, but to
characterize a program that uses 12 GB of RAM as "efficient" is about
as out of whack as it gets. That's the opposite of efficient.
Additionally, suggesting that using 12 GB of RAM "will make it operate
much faster", well there's simply no evidence to support that claim.

That's why I figured he was only being sarcastic, and not to be taken
seriously.

 

[VanguardLH]

I've just finished some testing here.

Setup:

Win7 Home Premium x64 OEM
Laptop with 320GB internal drive (C
3GB physical memory (two SODIMM)
External USB drive 110GB partition 3.2 million files (my testing partition)

I ran chkdsk by doing the old Properties:Tools tab thing and
selecting disk checking from there.

Using Resource Monitor, memory usage by chkdsk (inside Explorer) was
up around 2.7 or 2.8GB. It grew steadily until it ran out of
memory to grab (like it stopped at the remaining physical limit).
Page faults were close to zero, job finished about as quickly as
you might expect.

For this test, all other activity on the machine was quiet. Just
me watching Resource Monitor while the chkdsk run progressed.

*******

Test case #2

For this test, I simulate a user trying to work while chkdsk is
running. I open GIMP and create a 40000x3000 pixel blank image
using New from the menu. And start working within it. The image
is supposed to be around 1GB or so. Memory resident, is usually
less than that.

Well, now the gloves come off. Chkdsk doesn't relinquish any
memory (the "commit" stays at the top value, resident memory
drops, and fills again), and instead, it is making fresh requests
as the chkdsk run proceeds. I can't really work in GIMP, because
there are as many page faults against GIMP image editor, as
against chkdsk.

The long and the short of it, in this case, "nobody wins".
I can't actually work productively in the image editor, and
chkdsk isn't running quite as fast as before.

For most of the second test case, the disk light on the laptop
was on solid. That's due to the paging going on (thrashing between
the two programs). I can imagine running multiple copies of
chkdsk, to check separate physical drives, would end up with
similar behavior.

So the only test case where I got satisfactory results, was
running one copy of CHKDSK and walking away from the computer.
Since I was checking the external drive, it wasn't like I was
waiting on it to complete (C: was not involved).
I could have been editing an image from the C: drive, using GIMP,
but performance was so slow (lots of "not responding"), the machine
really couldn't be used for anything else.

*******

Now, if I do these things in WinXP, the machine behaves normally,
and I can actually multitask. Since the disk with the 3.2 million
files was prepped on the WinXP machine, I did actually try it out
over there as well. I could work normally, while the chkdsk was
working on the 3.2 million files.

As for where the 3.2 million files came from, I wrote a small
program to write out a set of files (took about five hours runtime).
I wanted lots of files, so the chkdsk run would last long enough
to observe memory usage.

*******

The Windows 7 design is a step backwards from the WinXP version of chkdsk.

Paul

So it looks like chkdsk doesn't roll back its physical memory commitment
(into the pagefile or by dumping or reducing tables) to make room for
other processes. That sucks.

Did you perform only a file integrity scan or did you do a low-level
sector test (with the /r command-line switch or enabling the "Scan for
and attempt recovery of bad sectors" option)? Testing for reliable
sector read/write incurs more overhead. See:

http://support.microsoft.com/kb/2264358

File integrity checking is, well, of some use but when I run chkdsk it's
always with the /r command-line switch. Because that includes sector
testing (on both allocated and unallocated sectors), it takes a long
time to complete and seems foolish to have files opened since the file
you're working on could be on a sector found bad.