When you run a large number of Windows-based programs, "Out Of Memory" error messages appear when you attempt to start new programs or try to use programs
that are already running.
In the case of WinNT services the process simply dies for no obvious reason.
This error can occur if the desktop heap in the WIN32 subsystem is depleted; even though you still have plenty of physical and pagefile memory available.
Desktop Heap memory is defined in the registry under the (long string) key 'SharedSection'::
Default settings Windows XP:
HKLM\System\CurrentControlSet\Control\Session Manager\SubSystems %SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,3072 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off MaxRequestThreads=16
In this case 1024 defines the system heap size, 3072 controls the size of the
interactive desktop heap (used for Windows objects).
If a third value is present e.g. "1024,3072,512", this covers the non-interactive desktop heap. All values are KBytes.
If the 3rd value is not present non-interactive desktops will default to the same value as interactive desktops (typically 3072 K).
In Vista SP1 or Windows Server 2008 this increases to 12 MB:
64-bit Windows has a default interactive desktop heap size of 20MB
Microsoft do not recommend that you set a value that is over 20,480 KB for the second SharedSection value.
Lower values increase performance at the risk of crashing any process which
can run out of resources.
For many applications (in Win XP) 3072K is too high, although MS recommend 512 K (in Q142676) for many applications this is too low.
512 K will support approximately 2,500 windows, menus, pens, icons, etc.
1024 K will support approximately 5,000 windows, menus, pens, icons, etc.
3072 K will support approximately 15,000 windows, menus, pens, icons, etc.
Because the non-interactive desktop heap is mapped into the address space of each and every process', this value should not be set to an arbitrarily high value, but should only be increased sufficiently to allow all the desired applications to run. Note that many server side applications such as Oracle Forms will spawn multiple processes for each user request.
Because NT only supports one interactive session at a time; tuning the interactive desktop heap has less impact on total memory use.
Windows NT has a system wide memory limit of 48 Mb for all the above desktop heaps, if this limit is exceeded there might not be enough memory to create an error message dialog box. As a result, the requested operation fails without any indication.
Desktop heap monitor - Examine desktop heap usage for WinXP/2003
Q126962 - Desktop Heap - NT 4, Windows 2000
Q184802 - Desktop Heap - NT 4, Windows 2000, Windows XP, 2003
Q947246 - Desktop Heap - Vista
Desktop Heap Overview - Blog from the Microsoft Critical Problem Resolution (CPR) Platforms Team.