Results 1 to 14 of 14
-
2012-11-27, 10:47 AM (ISO 8601)
- Join Date
- Dec 2006
- Location
- Raleigh NC
- Gender
Windows 7 standby memory question
I have a program which is generating a lot of write-only log files. Windows 7 is putting these log files in standby memory, with the result that standby memory is filling up with a bunch of useless stuff.
Is there a way to tell windows 7 NOT to cache those files in standby memory? Barring that, is there a way to minimize the time those files will spend in standby? To tell windows to treat those files like radioactive waste, and to get them out of standby memory as quickly as possible?
If absolute necessary, I suppose I could just disable standby memory , if that is possible. But I don't want to. It's far better that windows learn how to do its job better than it is to disable a core windows function.
Any ideas?
Respectfully,
Brian P."Every lie we tell incurs a debt to the truth. Sooner or later, that debt is paid."
-Valery Legasov in Chernobyl
-
2012-11-27, 10:05 PM (ISO 8601)
- Join Date
- Aug 2006
- Location
Re: Windows 7 standby memory question
Do you need the program?
If yes...why is it generating so many log files...and can you control the output of them?
As the take away from a couple of Google searches there appears to be a Microsoft tool that might help...but that is only from reading other posts from users that seem to have problems revolving around memory...which in some cases seems directed at Standby Memory
RAMMap
-
2012-11-28, 12:42 AM (ISO 8601)
- Join Date
- Apr 2010
- Location
- London, EU
- Gender
Re: Windows 7 standby memory question
Can you not redirect them to /Temp ?
π = 4
Consider a 5' radius blast: this affects 4 squares which have a circumference of 40' — Actually it's worse than that.
Completely Dysfunctional Handbook
Warped Druid Handbook
Avatar by Caravaggio
-
2012-11-28, 01:46 AM (ISO 8601)
- Join Date
- Feb 2007
- Location
- Manchester, UK
- Gender
Re: Windows 7 standby memory question
What exactly is "standby memory"? Been using computers for years and never come across that...are you talking about a ReadyBoost cache or something?
-
2012-11-28, 08:28 AM (ISO 8601)
- Join Date
- Dec 2006
- Location
- Raleigh NC
- Gender
Re: Windows 7 standby memory question
Behold .
Standby memory is something new for Windows Vista and Windows 7.
Wayy back in the day, there were two kinds of memory: Used memory and free memory.
But the smart folks at Microsoft noted how much of the memory is wasted on a typical computer. If you've got 4 GB of memory but you never use more than 2GB of it, why have the chips in your computer?
So that is where the idea of standby memory comes from. Windows Vista and Windows 7 aggressively cache memory in your computer. When a program finishes with an area of memory, it doesn't go back to free memory. Instead, it goes to standby memory. Windows assumes that if you used it recently, there's a very good chance you're going to use it again in the near future. So it keeps "hot" in standby memory so that, if the program wakes up and wants a similar area of memory again, Windows can simply give it back what it was previously using instead of going through the process of reallocating memory.
In theory, this makes the process of allocating memory much faster than having to free memory and then re-allocate it from scratch.
In practice , it doesn't seem to work that way in all cases. Certain applications, such as VM ware or games or databases written for older versions of windows, do their own memory management.
Case in point
example 2 . In this case, the guys' standby memory is filled with old zip files and that standby memory is never being released, resulting in errors.
The particular system I am having the problem with is IBM's DB2 . And yes, database logging is fundamental to the program and to disaster recovery.
Here's the way it works: It takes time for a database to store data in the raw storage area. So in order to speed up responsiveness, whenever a transaction is completed it is written to a .log file. In its own time, the database will make the appropriate modifications to the datafile and the .log file will become unnecessary.
So the state of the database is the database state plus the log files.
In theory, this allows you to restore your database up to the exact moment a disaster occurred. You simply restore your last backup, then "roll forward" the log files from the time of the last backup to present, and presto! All data recovered, nothing lost.
This is why, when I see the end of Star Ocean 3 (which takes place in a computer simulation), I say "Ah. The developers restored the database immediately prior to the point of Luther hitting the 'delete everything key'. This is why all the events up to that moment are preserved, but nothing before that moment is undone".
Seriously, fault-tolerant databases are constructed with disaster recovery to the second of disaster in mind in the modern world. Why would they not be in the 27th century, the time period in which that game is set?
At any rate, that is why DB2 generates lots and lots and lots of log files. It appears that Windows 7 is caching those log files in standby memory and not releasing them.
Of course, my first stop was the DB2 areas . No joy. Perhaps I'm barking up the wrong tree, or perhaps I'm one of the first people to notice this in relation to DB2.
DB2, like VMware, is essentially a virtual operating system. It performs its own memory and file management separate from the operating system and even the login permissions can be separated entirely from the OS. The root or administrator user for the computer may have no permissions whatsoever to access the database.
So it should come as no surprise that when an optimizing operating system collides with an optimizing program , errors of various kinds occur.
ETA: there is a good tool called RAMMAP . It allows you to purge standby memory in real time. The problem is, there's no way to stop standby memory from filling up with the log files again using this tool. So it's only a bandaid, not a solution.
It is not feasible to redirect the output of the logs to /temp. In the first place, /tmp is a Linux thing, not a windows thing. There IS a temporary directory on windows but writing to it would not solve the problem for two reasons:
1) Writing a permanent database log to a temporary directory defeats the purpose of disaster recovery.
2) The mere fact of creating the file means that memory must be allocated for it, and once that memory is finished with it will be relegated to standby memory. So regardless of whether the file is written to disk or to main memory pretending to be disk (as in the case of Linux' /tmp) , the standby memory will still be filled up.
That, at any rate, is where I am. I hope this has been useful!
Respectfully,
Brian P.Last edited by pendell; 2012-11-28 at 08:42 AM.
"Every lie we tell incurs a debt to the truth. Sooner or later, that debt is paid."
-Valery Legasov in Chernobyl
-
2012-11-28, 10:04 AM (ISO 8601)
- Join Date
- Apr 2010
- Location
- London, EU
- Gender
Re: Windows 7 standby memory question
Sorry - I thought you didn't want the data.
/Temp is a thing in Windows, albeit not useful in this case.
I'm assuming that the log files are being written to disk ?
All files in Windows are memory mapped, the standby memory is actually a very old trick to avoid paging in memory which is already available in memory. If the system needs more pages then these will simply be stolen and reused — so this should not be a problem.
I'm failing to see how this is a problem — its just an artefact of the virtual memory subsystem.π = 4
Consider a 5' radius blast: this affects 4 squares which have a circumference of 40' — Actually it's worse than that.
Completely Dysfunctional Handbook
Warped Druid Handbook
Avatar by Caravaggio
-
2012-11-28, 10:38 AM (ISO 8601)
- Join Date
- Dec 2006
- Location
- Raleigh NC
- Gender
Re: Windows 7 standby memory question
Noted.
I'm assuming that the log files are being written to disk ?
All files in Windows are memory mapped, the standby memory is actually a very old trick to avoid paging in memory which is already available in memory. If the system needs more pages then these will simply be stolen and reused — so this should not be a problem.
I'm failing to see how this is a problem — its just an artefact of the virtual memory subsystem.
Respectfully,
Brian P."Every lie we tell incurs a debt to the truth. Sooner or later, that debt is paid."
-Valery Legasov in Chernobyl
-
2012-11-28, 11:33 AM (ISO 8601)
- Join Date
- Apr 2010
- Location
- London, EU
- Gender
Re: Windows 7 standby memory question
I can see why you are suspicious, but these are different things. It is possible you have found a bug but the standby memory is not the same thing as your programs address space.
What is the size of your address space — I ask only because it is possible to increase this, if you already haven't ?
Can you roll the log files ? This should at least allow you to test your hypothesis.π = 4
Consider a 5' radius blast: this affects 4 squares which have a circumference of 40' — Actually it's worse than that.
Completely Dysfunctional Handbook
Warped Druid Handbook
Avatar by Caravaggio
-
2012-11-28, 12:02 PM (ISO 8601)
- Join Date
- Dec 2006
- Location
- Raleigh NC
- Gender
Re: Windows 7 standby memory question
Um .. I didn't know it was possible. I thought those limits were set by Microsoft. How would I find out what they are? The OS is server 2008 R2 and hasn't been modified from it's out of the box settings, if that's any help.
Can you roll the log files ? This should at least allow you to test your hypothesis.
Respectfully,
Brian P."Every lie we tell incurs a debt to the truth. Sooner or later, that debt is paid."
-Valery Legasov in Chernobyl
-
2012-11-28, 12:41 PM (ISO 8601)
- Join Date
- Apr 2010
- Location
- London, EU
- Gender
Re: Windows 7 standby memory question
The default is 2Gb for 32 bit apps, it requires a simple relink to change — but you might not have access to the source/object files.
You can verify this with the following API call
http://msdn.microsoft.com/en-us/libr...=vs.85%29.aspx
It's the ullTotalVirtual value you want.
You might be able to deduce this value from Perfmon ?
It's possible/likely that this has already been done?
I understand, but you should have a test system somewhere.π = 4
Consider a 5' radius blast: this affects 4 squares which have a circumference of 40' — Actually it's worse than that.
Completely Dysfunctional Handbook
Warped Druid Handbook
Avatar by Caravaggio
-
2012-11-28, 01:43 PM (ISO 8601)
- Join Date
- Dec 2006
- Location
- Raleigh NC
- Gender
Re: Windows 7 standby memory question
*Puzzled*
I can't make that call within DB2.
Our application is written in Java, so I'd have to write a JNI interface to call out and use that operation. I'll have to try to deduce from the performance monitor.
I understand, but you should have a test system somewhere.
Respectfully,
Brian P."Every lie we tell incurs a debt to the truth. Sooner or later, that debt is paid."
-Valery Legasov in Chernobyl
-
2012-11-28, 05:14 PM (ISO 8601)
- Join Date
- Feb 2007
- Location
- Manchester, UK
- Gender
Re: Windows 7 standby memory question
I have heard of that, yes, but I've never heard it referred to as "standby memory". It's the same disc cache that existed in prior versions of Windows, except they actually make the effort to pre-fill it with stuff they think you might need, rather than leaving it idle when the disc isn't actually being accessed by any applications.
nedz is quite correct when he says this has nothing to do with your application's virtual address space, though. Note that it's only possible to increase this beyond 2Gb for a 32-bit application if you're (a) running it on a 64-bit version of Windows or (b) set the /3GB parameter in the BOOT.INI file, and in case (b) you can only increase it to 3Gb--the other gigabyte is needed by the system.
-
2012-11-28, 05:42 PM (ISO 8601)
- Join Date
- Apr 2010
- Location
- London, EU
- Gender
Re: Windows 7 standby memory question
He is running on a 64 bit OS.
(I'm assuming here that the app is still 32 bit.)
On a 64 bit OS you can get almost the whole 4Mb because almost all of the OS stuff is above the 4Mb line. The App does need to be linked to use this though, this is the IMAGE_FILE_LARGE_ADDRESS_AWARE flag — you need this to generate addresses above 2Mb.Last edited by nedz; 2012-11-28 at 05:43 PM.
π = 4
Consider a 5' radius blast: this affects 4 squares which have a circumference of 40' — Actually it's worse than that.
Completely Dysfunctional Handbook
Warped Druid Handbook
Avatar by Caravaggio
-
2012-11-28, 05:58 PM (ISO 8601)
- Join Date
- Dec 2006
- Location
- Raleigh NC
- Gender
Re: Windows 7 standby memory question
He is running on a 64 bit OS.
(I'm assuming here that the app is still 32 bit.)
Respectfully,
Brian P."Every lie we tell incurs a debt to the truth. Sooner or later, that debt is paid."
-Valery Legasov in Chernobyl