When you run a large number of Windows-based programs, "Out Of Memory" error messages appear when you attempt to start new programs or try to use programs that are already running.
In current versions of Windows the tweaks described below are no longer needed - just use the defaults.
In the case of WinNT services the process simply dies for no obvious reason.
This error can occur if the desktop heap in the WIN32 subsystem is depleted; even though you still have plenty of physical and pagefile memory available.
Desktop Heap memory is defined in the registry under the (long string) key 'SharedSection'::
Default settings Windows XP:
HKLM\System\CurrentControlSet\Control\Session Manager\SubSystems %SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,3072 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off MaxRequestThreads=16
In this case 1024 defines the system heap size, 3072 controls the size of the
interactive desktop heap (used for Windows objects).
If a third value is present e.g. "1024,3072,512", this covers the non-interactive desktop heap. All values are KBytes.
If the 3rd value is not present non-interactive desktops will default to the same value as interactive desktops (typically 3072 K).
In Vista SP1 or Windows Server 2008 this increases to 12 MB:
64-bit Windows has a default interactive desktop heap size of 20MB
Lower values increase performance at the risk of crashing any process which can run out of resources.
Microsoft do not recommend that you set a value that is over 20,480 KB for the second SharedSection value.
Because the non-interactive desktop heap is mapped into the address space of
each and every process', this value should not be set to an arbitrarily high
value, but should only be increased sufficiently to allow all the desired applications
Many server applications will spawn multiple processes for each user request.
Microsoft recommend 512 K (in Q142676) but some applications may require more.
512 K will support approximately 2,500 windows, menus, pens, icons, etc.
1024 K will support approximately 5,000 windows, menus, pens, icons, etc.
3072 K will support approximately 15,000 windows, menus, pens, icons, etc.
Because Windows only supports one interactive session at a time; tuning the interactive desktop heap has less impact on total memory use.
Windows has a system wide memory limit of 48 Mb for all the above desktop heaps, if this limit is exceeded there might not be enough memory to create an error message dialog box. As a result, the requested operation fails without any indication.
Desktop heap monitor - Examine desktop heap usage for WinXP/2003
Q947246 - Desktop Heap - Vista
Desktop Heap Overview - Blog from the Microsoft Critical Problem Resolution (CPR) Platforms Team.