Windows 7 standby memory question

I have a program which is generating a lot of write-only log files. Windows 7 is putting these log files in standby memory, with the result that standby memory is filling up with a bunch of useless stuff.

Is there a way to tell windows 7 NOT to cache those files in standby memory? Barring that, is there a way to minimize the time those files will spend in standby? To tell windows to treat those files like radioactive waste, and to get them out of standby memory as quickly as possible?

If absolute necessary, I suppose I could just disable standby memory , if that is possible. But I don't want to. It's far better that windows learn how to do its job better than it is to disable a core windows function.

Any ideas?

Respectfully,

Brian P.

Do you need the program?

If yes...why is it generating so many log files...and can you control the output of them?

As the take away from a couple of Google searches there appears to be a Microsoft tool that might help...but that is only from reading other posts from users that seem to have problems revolving around memory...which in some cases seems directed at Standby Memory

RAMMap

Can you not redirect them to /Temp ?

What exactly is "standby memory"? Been using computers for years and never come across that...are you talking about a ReadyBoost cache or something?

Quote:

---

Originally Posted by factotum

What exactly is "standby memory"? Been using computers for years and never come across that...are you talking about a ReadyBoost cache or something?

---

Behold .

Standby memory is something new for Windows Vista and Windows 7.

Wayy back in the day, there were two kinds of memory: Used memory and free memory.

But the smart folks at Microsoft noted how much of the memory is wasted on a typical computer. If you've got 4 GB of memory but you never use more than 2GB of it, why have the chips in your computer?

So that is where the idea of standby memory comes from. Windows Vista and Windows 7 aggressively cache memory in your computer. When a program finishes with an area of memory, it doesn't go back to free memory. Instead, it goes to standby memory. Windows assumes that if you used it recently, there's a very good chance you're going to use it again in the near future. So it keeps "hot" in standby memory so that, if the program wakes up and wants a similar area of memory again, Windows can simply give it back what it was previously using instead of going through the process of reallocating memory.

In theory, this makes the process of allocating memory much faster than having to free memory and then re-allocate it from scratch.

In practice , it doesn't seem to work that way in all cases. Certain applications, such as VM ware or games or databases written for older versions of windows, do their own memory management.

Case in point

example 2 . In this case, the guys' standby memory is filled with old zip files and that standby memory is never being released, resulting in errors.

The particular system I am having the problem with is IBM's DB2 . And yes, database logging is fundamental to the program and to disaster recovery.

Here's the way it works: It takes time for a database to store data in the raw storage area. So in order to speed up responsiveness, whenever a transaction is completed it is written to a .log file. In its own time, the database will make the appropriate modifications to the datafile and the .log file will become unnecessary.

So the state of the database is the database state plus the log files.

In theory, this allows you to restore your database up to the exact moment a disaster occurred. You simply restore your last backup, then "roll forward" the log files from the time of the last backup to present, and presto! All data recovered, nothing lost.

This is why, when I see the end of Star Ocean 3 (which takes place in a computer simulation), I say "Ah. The developers restored the database immediately prior to the point of Luther hitting the 'delete everything key'. This is why all the events up to that moment are preserved, but nothing before that moment is undone".

Seriously, fault-tolerant databases are constructed with disaster recovery to the second of disaster in mind in the modern world. Why would they not be in the 27th century, the time period in which that game is set?


At any rate, that is why DB2 generates lots and lots and lots of log files. It appears that Windows 7 is caching those log files in standby memory and not releasing them.

Of course, my first stop was the DB2 areas . No joy. Perhaps I'm barking up the wrong tree, or perhaps I'm one of the first people to notice this in relation to DB2.

DB2, like VMware, is essentially a virtual operating system. It performs its own memory and file management separate from the operating system and even the login permissions can be separated entirely from the OS. The root or administrator user for the computer may have no permissions whatsoever to access the database.

So it should come as no surprise that when an optimizing operating system collides with an optimizing program , errors of various kinds occur.

ETA: there is a good tool called RAMMAP . It allows you to purge standby memory in real time. The problem is, there's no way to stop standby memory from filling up with the log files again using this tool. So it's only a bandaid, not a solution.


It is not feasible to redirect the output of the logs to /temp. In the first place, /tmp is a Linux thing, not a windows thing. There IS a temporary directory on windows but writing to it would not solve the problem for two reasons:

1) Writing a permanent database log to a temporary directory defeats the purpose of disaster recovery.

2) The mere fact of creating the file means that memory must be allocated for it, and once that memory is finished with it will be relegated to standby memory. So regardless of whether the file is written to disk or to main memory pretending to be disk (as in the case of Linux' /tmp) , the standby memory will still be filled up.

That, at any rate, is where I am. I hope this has been useful!

Respectfully,

Brian P.

Sorry - I thought you didn't want the data.
/Temp is a thing in Windows, albeit not useful in this case.

I'm assuming that the log files are being written to disk ?

All files in Windows are memory mapped, the standby memory is actually a very old trick to avoid paging in memory which is already available in memory. If the system needs more pages then these will simply be stolen and reused — so this should not be a problem.

I'm failing to see how this is a problem — its just an artefact of the virtual memory subsystem.

Quote:

---

Originally Posted by nedz

Sorry - I thought you didn't want the data.
/Temp is a thing in Windows, albeit not useful in this case.

---

Noted.

Quote:

---

I'm assuming that the log files are being written to disk ?

---

Yes.

Quote:

---

All files in Windows are memory mapped, the standby memory is actually a very old trick to avoid paging in memory which is already available in memory. If the system needs more pages then these will simply be stolen and reused — so this should not be a problem.

I'm failing to see how this is a problem — its just an artefact of the virtual memory subsystem.

---

That's certainly the theory. But as in the case of the VMware and the other example I listed, it certainly appears that standby memory is not being released as it should be. Unfortunately, I'm at a loss to verify or disprove this. Certainly there are lots of windows people who will insist that this is not a problem. And it IS possible that I am misguided. But I am at this point unable to advance an alternate hypothesis as to why a sudden lack of memory for my application should coincide with a filling up of standby memory with these files.

Respectfully,

Brian P.

Quote:

---

Originally Posted by pendell

That's certainly the theory. But as in the case of the VMware and the other example I listed, it certainly appears that standby memory is not being released as it should be. Unfortunately, I'm at a loss to verify or disprove this. Certainly there are lots of windows people who will insist that this is not a problem. And it IS possible that I am misguided. But I am at this point unable to advance an alternate hypothesis as to why a sudden lack of memory for my application should coincide with a filling up of standby memory with these files.

---

I can see why you are suspicious, but these are different things. It is possible you have found a bug but the standby memory is not the same thing as your programs address space.
What is the size of your address space — I ask only because it is possible to increase this, if you already haven't ?
Can you roll the log files ? This should at least allow you to test your hypothesis.

Quote:

---

Originally Posted by nedz

I can see why you are suspicious, but these are different things. It is possible you have found a bug but the standby memory is not the same thing as your programs address space.
What is the size of your address space — I ask only because it is possible to increase this, if you already haven't ?

---

Um .. I didn't know it was possible. I thought those limits were set by Microsoft. How would I find out what they are? The OS is server 2008 R2 and hasn't been modified from it's out of the box settings, if that's any help.

Quote:

---

Can you roll the log files ? This should at least allow you to test your hypothesis.

---

While it is possible in theory to do so, I can't do it on this production system. If we find a research system to play with, it might be worth exploring.

Respectfully,

Brian P.

Quote:

---

Originally Posted by pendell

Um .. I didn't know it was possible. I thought those limits were set by Microsoft. How would I find out what they are? The OS is server 2008 R2 and hasn't been modified from it's out of the box settings, if that's any help.

---

The default is 2Gb for 32 bit apps, it requires a simple relink to change — but you might not have access to the source/object files.
You can verify this with the following API call
http://msdn.microsoft.com/en-us/libr...=vs.85%29.aspx
It's the ullTotalVirtual value you want.
You might be able to deduce this value from Perfmon ?
It's possible/likely that this has already been done?

Quote:

---

Originally Posted by pendell

While it is possible in theory to do so, I can't do it on this production system. If we find a research system to play with, it might be worth exploring.

---

I understand, but you should have a test system somewhere.

Quote:

---

Originally Posted by nedz

The default is 2Gb for 32 bit apps, it requires a simple relink to change — but you might not have access to the source/object files.
You can verify this with the following API call
http://msdn.microsoft.com/en-us/libr...=vs.85%29.aspx
It's the ullTotalVirtual value you want.
You might be able to deduce this value from Perfmon ?
It's possible/likely that this has already been done?

---

*Puzzled*

I can't make that call within DB2.
Our application is written in Java, so I'd have to write a JNI interface to call out and use that operation. I'll have to try to deduce from the performance monitor.

Quote:

---

I understand, but you should have a test system somewhere.

---

Indeed. In R&D we have many test systems, and they are all currently in use, either for testing different aspects of this problem or other problems on other systems. But when the list of higher priority problems shortens and the number of available systems increases -- then perhaps. If the problem isn't OBE or one of the other solutions found works. My boss has found what he believes is a solution to the problem and it is being uploaded onto a test system as we speak. If it works , great. If it doesn't, we work down the list of other things to explore.

Respectfully,

Brian P.

I have heard of that, yes, but I've never heard it referred to as "standby memory". It's the same disc cache that existed in prior versions of Windows, except they actually make the effort to pre-fill it with stuff they think you might need, rather than leaving it idle when the disc isn't actually being accessed by any applications.

nedz is quite correct when he says this has nothing to do with your application's virtual address space, though. Note that it's only possible to increase this beyond 2Gb for a 32-bit application if you're (a) running it on a 64-bit version of Windows or (b) set the /3GB parameter in the BOOT.INI file, and in case (b) you can only increase it to 3Gb--the other gigabyte is needed by the system.

He is running on a 64 bit OS.

(I'm assuming here that the app is still 32 bit.)
On a 64 bit OS you can get almost the whole 4Mb because almost all of the OS stuff is above the 4Mb line. The App does need to be linked to use this though, this is the IMAGE_FILE_LARGE_ADDRESS_AWARE flag — you need this to generate addresses above 2Mb.

Quote:

---

He is running on a 64 bit OS.

(I'm assuming here that the app is still 32 bit.)

---

Correct. Both the application and the underlying DB2 are 32 bit apps running on 64 bit Windows 2008 .

Respectfully,

Brian P.