Search results matching tag 'Debugging'

Troubleshooting performance counter problems with perfmon and a debugger

Anonymous — Sun, 24 May 2009 04:17:00 GMT

Performance counters are a wonderful tool when it comes to quickly assessing system health and performance of Windows subsystems. In addition to learning much about the operation of Microsoft products, exposing performance counters from your own code...(read more)

How To View what ASP.NET Requests are doing at runtime on IIS 6.0

Anonymous — Tue, 05 May 2009 18:29:00 GMT

This is just a quick blog to mention a forgotten tool. The managed stack explorer can be run on an IIS 6.0 Server running ASP.NET 2.0 to investigate the managed call stacks. Looking at the call stacks when an ASP.NET application is not responding may help identify what the requests are doing.

Setting up the tool

Download the Managed Stack Explorer from here:

http://www.microsoft.com/downloads/details.aspx?FamilyID=80cf81f7-d710-47e3-8b95-5a6555a230c2&displaylang=en

Run the MSI and install to a directory of your choice. The default location is C:\Program Files\Power Toys for Visual Studio\Managed Stack Explorer

How the Tool Works

The Managed Stack Explorer contains a User Interface that allows you to choose a process and thread and see the managed call stack at that point in time. See below:

The UI also gives a quick glimpse at the % time a process has spent in the GC. As a general rule, if that number is > 10-15% it might indicate a performance problem. For more information on monitoring ASP.NET Performance see this whitepaper: http://msdn.microsoft.com/en-us/library/ms972959.aspx

The Managed stack explorer also has a stack logging feature that can log the stack trace over time. When using the UI it takes a snapshot whenever a thread is selected in the thread window. It takes a snapshot of only this thread at that point in time. It does this by attaching to the process, collecting information about the thread and then detaching. When using the stack logging, the tool can be configured to scan the managed threads of a single process at a specified interval and log that to a file.

Using the stack Logging feature

For most cases where someone will want to use this tool, the stack logging feature is a good idea. IIS Server administrators might get notified that their ASP.NET Application is not responding, or the site is slow. When this happens the Managed Stack Explorer can be run with stack logging enabled for a period of time, then the log analyzed to determine what the threads and ASP.NET Requests were doing during the period of poor performance.

The following steps show how to configure Managed Stack Explorer with stack logging.

1) Choose Options->Preferences and configure the stack logging rate. The default is to log the call stack every 5 seconds. The other settings only apply to the auto refresh of the User interface.

2) Determine the Process ID Of the application to monitor. The Managed Stack Explorer shows the processes that have the .NET runtime loaded in the left pane. When multiple W3WP.exe’s are running use the following command to determine the Application Pools process ID.

a. Cscript.exe %systemroot%\system32\iisapp.vbs

3) In the processes window double-click the w3wp.exe in which the ASP.NET Application is running. This will cause the thread window to populate with a list of managed threads in that process.

4) In the Thread list window select all threads, by selecting the first thread, holding shift and then selecting the last thread in the list. The stack logging will only monitor the selected threads, so make sure to select all of them.

5) Choose Action->Start Stack Logging from the menu and specify a file to log the stack traces too. This will start the stack logging

6) Stop the trace after enough data is collected and then choose Action->Stop Stack Logging from the menu.

7) Open the generated log file and look at what the threads were doing.

Analyzing the stack log file

Open the generated log file in notepad or another text editor. You will get output similar to the following for each thread:

4/28/2009 11:31:14 AM

Stack trace depth is set to show all frames

Thread ID: 5196

0. _Default.Page_Load (Source Unavailable)

1. System.Web.Util.CalliHelper.EventArgFunctionCaller (Source Unavailable)

2. System.Web.Util.CalliEventHandlerDelegateProxy.Callback (Source Unavailable)

3. System.Web.UI.Control.OnLoad (Source Unavailable)

4. System.Web.UI.Control.LoadRecursive (Source Unavailable)

5. System.Web.UI.Page.ProcessRequestMain (Source Unavailable)

6. System.Web.UI.Page.ProcessRequest (Source Unavailable)

7. System.Web.UI.Page.ProcessRequest (Source Unavailable)

8. System.Web.UI.Page.ProcessRequestWithNoAssert (Source Unavailable)

9. System.Web.UI.Page.ProcessRequest (Source Unavailable)

10. ASP.default_aspx.ProcessRequest (Source Unavailable)

11. System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute (Source Unavailable)

12. System.Web.HttpApplication.ExecuteStep (Source Unavailable)

13. System.Web.HttpApplication.ApplicationStepManager.ResumeSteps (Source Unavailable)

14. System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest (Source Unavailable)

15. System.Web.HttpRuntime.ProcessRequestInternal (Source Unavailable)

16. System.Web.HttpRuntime.ProcessRequestNoDemand (Source Unavailable)

17. System.Web.Hosting.ISAPIRuntime.ProcessRequest (Source Unavailable)

The key pieces are the Thread ID and the Date/Time. Look for instances where the same thread ID is in the same call for more than a couple of intervals. In the above stack we can see that it is currently executing the Default.aspx Page_Load routine. If this stack is identical at our configured 5 second intervals, then this would indicate that something in the Default.aspx page’s Page_load routine is taking greater than 5-10 seconds. Therefore, that routine should be investigated.

Depending on the application, the call stacks may be different. The stacks may indicate a connectivity problem to a backend database, call to a web service, or something else. Either way the Managed Stack explorer can be used to quickly collect data that may be helpful before using more invasive tools. And the information generated can help developers of the application understand what the application is doing when it is performing slowly.

If the output does not show any activity on any of the threads, then another form of troubleshooting might be needed. Some causes of the threads being empty could be:

1) The request finished before stack logging started

2) The request is being handled by an Asynchronous handler, and it hasn’t notified ASP.NET that the request has finished.

3) The request has not been delivered to ASP.NET yet and is still running on a native thread.

Conclusions

The Managed Stack Explorer is a tool that can be used to view what .NET call stacks are doing at runtime. It does this by attaching to the process and taking a snapshot of the thread and displays it in the UI or logs it to a file. The managed stack can be used to determine what an ASP.NET request is doing at the time of the snapshot, and by capturing snapshots over a period of time, where the ASP.NET Request might be running slowly, it can be used to help identify what the bottleneck may be.

Lastly, this tool also has source code available for anyone that might want to extend its features. Please see the following site for information on how to do that.

http://www.codeplex.com/MSE

What do you need to troubleshoot Azure?

Anonymous — Wed, 19 Nov 2008 11:00:00 GMT

Looking to the future with cloud computing, it is going to become increasingly important to have good information about what is happening with your site in order to properly maintain it.

Keeping development type of issues aside, what types of things do you think you will need in order to be able to properly troubleshoot a problem in your application once it is deployed to the cloud.

For example, when you publish your application, it will be similar to uploading today to a hoster in that:

No access to the server
Cannot get dumps of any processes
Cannot access perfmon logs
Cannot access the Event log

So given that type of situation:

What are the possible problem situations that may arise?
What types of information do you think you will require to find out what is going wrong with your application?
Would you try to pull your application onto your development box to try to repro some of the problems that way?

Or are there any other concerns or thoughts you have around this? I’d love to hear what you have to say.

Strange callstacks

Anonymous — Wed, 29 Oct 2008 10:00:00 GMT

How many times have you been troubleshooting a dump or application, you look at the callstack and you see something that just doesn’t quite look right. Chances are the problem is that you don’t have correct symbols.

For example, if you see something like this:

Vswebdesign!DllCanUnloadNow+0xb02a0
Vswebdesign!DllCanUnloadNow+0xb8ef4
Vswebdesign!DllCanUnloadNow+0xb2203
Vswebdesign!DllCanUnloadNow+0xb248f
Vswebdesign!DllCanUnloadNow+0x2c406
Vswebdesign!DllCanUnloadNow+105a89
...

That usually is pointing to this problem. You can see that the offsets for the functions are very large. Also, the callstack is the same function repeated over and over. And most applications probably don’t call DllCanUnloadNow.

If you get this problem, you can try to get the correct symbols, or just understand what you are seeing here and know that you aren’t seeing the correct functions. For most things that concern you, you should be able to get the correct symbols so that you can properly resolve your callstacks.

ASP.NET Tip: How to avoid creating a GC Hole

Anonymous — Thu, 25 Sep 2008 10:00:00 GMT

There are only a few things that can make a .NET process crash. The most common one is an Unhandled Exception getting raised. Another way that is can happen is by creating a GC Hole.

What is a GC Hole

So first a little background on what I mean by a GC Hole. A GC Hole is any corruption that happens inside of the managed heaps. Under normal circumstances, this cannot happen as you don’t have pointers that reference objects in the heap so you can’t corrupt them. This corruption is generally seen by the GC (Garbage Collector) when it is trying to compact the heaps and release objects no longer referenced.

How does a GC Hole get created

So if we don’t have access to pointers (not counting managed C++ and that is a whole different conversation), how can this occur? Well the most common way is by making a native call (P/Invoke) that returns data. There are actually two things that can happen during this process to cause a GC Hole.

The first is a buffer overrun. You can imagine passing a byte array to a native function and having that array be 200 bytes. If the native function returns 400 bytes worth of data, it will write past the end of the object and corrupt the next object in the managed heap.

The second happens if the object being passed to a native function isn’t pinned first.

Pinning is a whole topic on it’s own. There are lots of problems that can happen from pinning object in memory, a great deal of information can be found here on the most common problem you can have while pinning.

Back to the GC Hole issue. If the following events occur, it will cause a GC Hole:

Allocate a byte array to be passed to a native call, do not pin it
Make the native call passing the byte array
While the native call is processing the request, a Garbage Collection occurs
Native call returns

So what is the problem with the GC running before the native call returns? Well, the GC has the ability to compact the heap (move object around so they are next to each other) and if it moves our byte array to a new location, the native call will still have the old location as to where to write it’s data to. So when it returns, it will write the result into memory where the byte array used to be, thus corrupting anything in that memory location currently.

For example, If we had this when making the native call:

_________________________________________
| | | | | | | int | int | Free | byte | Free |
| | | | | |
_________________________________________

Then after the GC runs, it will compact the ints and byte next to each other like:

_________________________________________
| | | | | | int | int | byte | Free |
| | | | |
_________________________________________

Now assume objects get created inside that free block, we would then have something like:

_________________________________________
| | | | | | | | int | int | byte | String | String | Free |
| | | | | | |
_________________________________________

Then when the native calls returns, it will write to the original location of the byte array and thus corrupt the two strings.

How to find a GC Hole and fix it

This is something I will address shortly in another blog post. We will look into troubleshooting a GC Hole and see what we can do to find the problem.

I hope that this post will help developers understand the areas where they need to be careful and help to keep everyone from creating them in the first place. For information on how to pin an object correctly, take a look at GCHandles, Boxing and Heap Corruption.

More great information can be found at, Asynchronous operations, pinning

Who uses SOSEX and what for?

Anonymous — Mon, 22 Sep 2008 10:00:00 GMT

Another thing I am curious to know is if anyone is using SOSEX and what advantages it gives you for debugging. I have talked to the author of it and I think it has a lot of useful commands, just wanted to get everyone’s take on it.

Let me know any problems you have with it also.

If you don’t know what SOSEX is, take a look at: Steve’s blog

High Memory part 6 – Fragmentation revisited

Anonymous — Wed, 17 Sep 2008 10:00:00 GMT

I have talked about a bunch of commands used to troubleshoot a managed memory problem in the past and given some situations of using them like:

There is another way that we could see a problem that involves fragmentation. Let’s consider the situation where we see the same situation we saw before, namely:

But we do see a bunch of files listed when we look in lm. So what should this tell us about the problem? Well, first off, since the files are showing up in lm, we know the problem isn’t the same as, Dynamic Assemblies and what to do about them. So next we would want to look and see specifically what the files are.

Situation 1

The first common problem that you will see is caused by either have debug=true, or batch=false is set. Batch is similar to debug, set in the same place and will tell the compiler to not create one DLL for the entire directory but rather create one DLL for each file we compile. Underneath, debug=true sets batch=false, so they do the same thing in this regard. If that is the problem, you will see a bunch of strangely named DLLs in the lm list. Names like:

  hbdp25u7.dll
  1gxdnryw.dll
  fzgq0g7y.dll
  fhlpqzqm.dll
  su0utu5-.dll
  vhaewyii.dll
  tdkuail7.dll

Turning debug off, or batch on will resolve the problem in this situation.

Situation 2

The second common problem that will cause this is if you have a common framework that you are using in multiple applications. One common way to do this is the copy the framework files into the bin directory of each application. This will cause us to create a different in-memory file for each one. So instead of just having 5 files that make up your framework loaded in memory, you will have 5 * (# of AppDomains) loaded.

You can identify this problem because in the list of files, you will see the same filename repeated, only some will have some numbers after them:

  MyFramework.dll
  MyFramework_50008012.dll
  MyFramework_50002008.dll
  MyFramework_40101008.dll

This is just showing the different versions of the file by showing you the other copies with their base address (load address).

The easiest solution to this type of problem is to place the framework files into the GAC. Then you will only have one copy of the files which all the Applications will share.

Debugger Extension update

Anonymous — Fri, 12 Sep 2008 10:00:09 GMT

The debugger extension that we were working on getting out with the debugger package has hit some roadblocks and it looks like it isn't going to be shipping with the debugger anytime soon.

We are looking to see if we can find an alternative method to get it out. But for the meantime, just stay tuned and I'll let you know when I find something out.

I know the additional functions will be a lot of help to everyone that uses it so hopefully we can get something to you soon. I'll keep this blog updated with the latest.

How do you test the performance of your web site?

Anonymous — Wed, 10 Sep 2008 10:00:00 GMT

I wanted to get a feel for what tools people are using to test the performance of their website. My questions are:

Do you check page-load times?
Do you dig in to see which files are taking the time
How do you track down a page that uses a lot of memory
How do you track down a high CPU consuming page

I don’t want to list any possible tools as I’d like to hear what people are using without me putting any suggestions out.

If I don’t see some being mentioned, I’ll be sure to bring it up so we have a holistic view of the tools.

Look forward to seeing what people use.

SOS Tip: Using help

Anonymous — Tue, 09 Sep 2008 10:32:00 GMT

This is just a quick little note to remind everyone that the !help command in sos can be very helpful. Not only does it list all of the possible commands, but if you run it and pass it the name of a command, it will print out a bunch of really useful information for that command.

For example, if you look at the help for !dumpheap: !help dumpheap

You will see, for example, the syntax that you can use to pass the list of objects returned from !dumpheap -mt to a .foreach command.

I use the help a lot to check on the different switches and also for reminders like with .foreach, when to use a ( and when to use a {. Those always seem to trip me up.

Another really useful command inside help is the FAQ. If you run !help faq, it will print out all sorts of things that are really useful to know.

If you haven't checked out the !help command much in sos, please do now and let me know what you think.