High Memory due to System.WeakReference

We recently saw an issue that was manifesting as very high memory utilization by a w3wp.exe process that was hosting an ASP.net application.  To start troubleshooting, we gathered a memory dump of the process when its memory usage was very high.

A standard way of starting out the debugging of a high memory problem in a managed application is to run the !dumpheap –stat command.  When we did so, we found millions of 16 byte System.WeakReference objects (I’ve truncated the output for brevity’s sake):

0:000> !dumpheap -stat

Loading the heap objects into our cache.

total 28,603,114 objects

Statistics:

        MT    Count    TotalSize Class Name

0x648efc44    4,199       83,980 System.Configuration.ConfigurationValue

0x6639b220    4,368       87,360 System.Web.VirtualPath

0x6639e3fc    4,534       90,680 System.Web.Caching.CacheKey

0x663b4074    2,063       90,772 System.Web.HttpCookie

0x648ea704    2,003       96,144 System.Configuration.ConfigurationValues

0x79131840       99      104,612 System.DateTime[]

0x663b27f4    2,066      107,432 System.Web.HttpValueCollection

0x663aa5e8    2,072      107,744 System.Web.HttpModuleCollection

0x66435948    2,073      107,796 System.Collections.Generic.Dictionary`2

0x65409d70    1,772      113,408 System.Data.SimpleType

0x663ada24    4,126      115,528 System.Web.Hosting.RecyclableCharBuffer

0x663aa6f8    4,126      132,032 System.Web.HttpAsyncResult

0x7910d7e8    4,148      132,736 System.AsyncCallback

0x6641f33c    3,028      133,232 System.Web.UI.Control+OccasionalFields

0x65405fdc    1,102      141,056 System.Data.SqlClient._SqlMetaData

0x7a766474    8,912      142,592 System.ComponentModel.CollectionChangeEventArgs

0x79111038    2,748      153,888 System.Reflection.RuntimePropertyInfo

0x7910efbc   13,608      163,296 System.Runtime.Remoting.Messaging.CallContextSecurityData

0x663af054    2,063      165,040 System.Web.HttpWriter

0x653fe4d4    2,080      208,000 System.Data.SqlClient.SqlParameter

0x663b08c8    4,126      231,056 System.Web.HttpCookieCollection

0x02525e84    2,073      240,468 ASP._global_asax

0x79109778    4,395      246,120 System.Reflection.RuntimeMethodInfo

0x663af964    2,072      248,640 System.Web.SessionState.SessionStateModule

0x79102290   25,426      305,112 System.Int32

0x663aa0a0    2,063      354,836 System.Web.HttpRequest

0x663a9dac    2,063      387,844 System.Web.HttpContext

0x663a4adc    2,063      404,348 System.Web.HttpResponse

0x79108ce0   11,536      415,296 System.Collections.Hashtable+HashtableEnumerator

0x7911a2d0   13,610      489,960 System.Runtime.Remoting.Messaging.LogicalCallContext

0x79104de8   13,616      490,176 System.Threading.ExecutionContext

0x663b205c   31,080      497,280 System.Web.HttpApplication+SyncEventExecutionStep

0x663ad7c0    2,063      503,372 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6

0x7912d7c0    7,403      522,620 System.Int32[]

0x7a7566e8   26,959      539,180 System.ComponentModel.EventHandlerList+ListEntry

0x65412bb4      514      539,672 System.Data.RBTree`1+Node[[System.Data.DataRow, System.Data]][]

0x7a75a878   35,598      569,568 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry

0x79104368   25,337      608,088 System.Collections.ArrayList

0x0252788c    2,027      770,260 ASP.default_aspx

0x654088b4    5,281      781,588 System.Data.DataColumn

0x79108afc   11,541      877,116 System.Threading.ExecutionContext+ExecutionContextRunData

0x654359c8    1,038    1,125,544 System.Data.RBTree`1+Node[[System.Int32, mscorlib]][]

0x66437650    2,073    1,252,092 System.Collections.Generic.Dictionary`2+Entry

0x791044dc   53,803    1,721,696 System.EventHandler

0x79101fe4   31,106    1,741,936 System.Collections.Hashtable

0x000e2dc0    3,644    3,621,108      Free

0x7912d9bc   31,396    4,984,728 System.Collections.Hashtable+bucket[]

0x790fd8c4   95,552    9,502,548 System.String

0x7912dd40    8,376   12,904,904 System.Char[]

0x7912d8f8   53,376  336,430,944 System.Object[]

0x79104c38 27,851,257  445,620,112 System.WeakReference

Total 28,603,114 objects, Total size: 836,223,424

NOTE: For more information on the !dumpheap debugger command, have a look at the following blog post:

http://blogs.msdn.com/tess/archive/2005/11/25/496973.aspx

The next step was to dump out some of the System.WeakReference instances to try to dig in to why they’re there:

0:000> !dumpheap -mt 0x79104c38

Using our cache to search the heap.

   Address         MT     Size  Gen

0x025d7ab8 0x79104c38       16    2 System.WeakReference

0x025dc2d0 0x79104c38       16    2 System.WeakReference

0x025dc540 0x79104c38       16    2 System.WeakReference

0x025dd330 0x79104c38       16    2 System.WeakReference

0x025dd8d8 0x79104c38       16    2 System.WeakReference

0x025def80 0x79104c38       16    2 System.WeakReference

0x025def90 0x79104c38       16    2 System.WeakReference

0x025defa0 0x79104c38       16    2 System.WeakReference

0x025defb0 0x79104c38       16    2 System.WeakReference

0x025defc0 0x79104c38       16    2 System.WeakReference

0x025defd0 0x79104c38       16    2 System.WeakReference

0x025defe0 0x79104c38       16    2 System.WeakReference

0:000> !do 0x025defa0

Name: System.WeakReference

MethodTable: 79104c38

EEClass: 79104bd4

Size: 16(0x10) bytes

GC Generation: 2

 (C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)

Fields:

      MT    Field   Offset                 Type VT     Attr    Value Name

791016bc  40005a9        4        System.IntPtr  1 instance 33389776 m_handle

7910be50  40005aa        8       System.Boolean  1 instance        0 m_IsLongReference

All of the objects we dumped out looked the same, and none of them appeared to be rooted.  Looking around for other signs of problems, we found an components built in debug mode:

0:000> !sos.FindDebugModules

Loading all modules.

Searching for modules built in debug mode...

custom_component.DLL not built release

custom_component2.DLL not built release

custom_component3.DLL not built release

Having modules built in debug mode running on a Production server is never a good idea.  And as it turned out in this case, the debug mode modules combined with the fact that these modules implement the __ENCLIST helper class for Visual Studio’s Edit and Continue feature.

When a component is built in debug mode, the  Edit and Continue debugging feature of Visual Studio is enabled.  The Edit and Continue debugging feature in Visual Studio 2005 and 2008 maintains a list of weak references to objects that have been created. If the object that has been created is a class that contains an event, these weak references are maintained for the duration of the program. This behavior increases memory usage.  To run into the memory usage problem, the component must be built in DEBUG mode, and in must use the __ENCLIST help class that allows edit and continue.

In this scenario, the problem got resolved by rebuilding those components in Release mode instead of Debug mode.