Sukesh's IIS Blog

DebugDiag : Introduction

When I was a developer (I mean employed as a developer, even now I develop applications, don’t get me wrongJ) before joining Microsoft, I used to get stuck with application issues; whether its process crashing, process not responding or high memory usage. There were not many options for me at that time but to review code for that page or form and figure out the cause myself.

If there was a tool which I can use to tear the process and see what is going on inside the process I would've saved a lot of my hours (or maybe days). Although Windows Debugging Tools (WinDBG) was available, it was not too easy to learn the commands or understand how all those stuff works when you have development timelines/deadlines (or is it death lines?) to be met.

Not many developers are aware of what kind debugging I’m gonna talk about. When ever we talk about debugging people assume that it’s live debugging using Visual Studio or so and put breakpoint and walking through the code. Huh! This would be a blessing if we can do the same thing in case of production servers, but on production server applications it’s a completely different ball game.

Think of a situation where customer has an issue on production box with IIS process

(Here IIS process is used just for illustration but below explained issues are true with any other multi-threaded process/service)

My Options (being little bit sarcastic)

1. Send windows source code to customer and we will install Visual Studio to walk through the code and find what the issue is” You know I would loose my job. J
   This kind of debugging works mostly for client applications
2. What happens if I don’t really know when the issue happens?
   I can employ a person who will sit in front of the server 24x7 watching for the issue to happen J
3. What if the issue happens only for sometime and the issue vanishes?
   By the time my monitoring person yawns, the issue would vanish J
4. What happens if the issue only happens when a specific user sends a post request with some specific string in there which leads the IIS process to get stuck and block all requests?
   Start writing “Debug.Print” or “Response.Write” kind of tracing to find out where it gets stuck. You might not finish your project anywhere in near future J
5. Customer called on my mobile and is screaming because my website is not responding.
   I need to run to the server (maybe even drive, since the server is in a remote place) to take a memory dumps.

All the above options are provided to understand how tools like DebugDiag help us to automate and make our life far better.

DebugDiag or Debug Diagnostic Tool is not the 1^st tool but as far as I know would be the 5^th generation of tool for doing post-mortem memory dump analysis. Most of those previous tools were were exclusively used by PSS and were not available on Microsoft Downloads.

Post mortem debugging simply means that we take a snapshot of the process memory when the issue happens and use either DebugDiag or WinDBG to figure out what was going on inside the process when the issue happened and find out the cause for the issue. Since this is technically challenging, it takes a lot of time. Some of our customers think that it's like looking into iislogs to find the request. Let me tell you that its mostly digging deep into thread stack, heap and other memory areas to find out what might’ve lead to the issue.

What is Debug Diagnostic Tool?

DebugDiag is a post mortem debugging tool which has analysis capabilities, so in simple words there is 3 parts for this tool.

1. Capture memory dumps for different types of issues (Hang/Crash/Memory)
2. Run basic analysis on the captured dumps and generates a report to understand the results. It also provides very good pointers to issues mostly for expert eyes.
3. Exposes an object model which can be used to easily access the information available inside the memory dump file (memory dump file extensions are usually DMP / PDMP / MDMP)

What are the main components?

1. Debug Diagnostic Service (dbgsvc.exe)
2. DebugDiag UI (debugdiag.exe)
3. DebugDiag Host (dbghost.exe)

Debug Diagnostic Service

This is the service which is the heart of DebugDiag. Why should it be a service? In the past we used AD+ (Auto Dump+) for troubleshooting most of those debug scenarios. AD+ is executed from the command line and client side program. This simply means that it runs under the context of the logged-in interactive user.

Let’s take an example. Assume that we are trying to track an issue which happens intermittently, say for example process crash and it happens once in a week or month.

So we setup AD+ (KB 286350) from command prompt. Since this tool runs from command line if you logout from the console AD+ stops monitoring. So if your organization has multiple administrators who look after the server they need to be informed not to logout from the console till we track and get a good set of dumps for the issue. This becomes extremely difficult specially because we find out that someone did a logout only after the next issue occurrence and by then its too late. Then we start monitoring again and sleep till we get another repro. Keep in mind, in some cases a repro might take seconds, minutes, hours, days, weeks or even months.

Another issue with AD+ like tools is that you cannot use it through Terminal Service sessions which most of those administrators are too used to J

AD+ provides a lot of customization options and its powerful in that way, and it was “the” tool we used in the past (and I see people using it even now).

To get around the above mentioned issues DebugDiag runs as a service as “Local System” so that it’s not dependent on the logged on interactive user session.

So how do we configure this service since windows services cannot have UI?

DebugDiag UI

DebugDiag user interface is used to create rules for capturing different types of issues by creating rules and also the interface to run the analysis portion of the tool.

DebugDiag like I mentioned before (did I mention?), has a scripting host built-in using which we can customize and extend the features according to the requirements. The main script file called “DbgSVC.vbs” (we call it as controller script) is present in scripts folder inside installation folder. This script gets modified when you make changes in the UI related to Hang or Memory Leak rules.  This script file contains (or exposes) some events which you can use to extend and customize the working of DebugDiag.

Open the Controller Script (“DbgSVC.vbs”) in notepad and see for yourself.

Rules are nothing but simple way of configuring DebugDiag to work according to your requirements for specific scenarios. Rules contains information about the location where you want the memory dumps files to be stored etc and also contains

Events you can further use. For example if you create Crash Rule, DebugDiag creates a script file called “CrashRule_IIS.vbs” in the scripts folder.

Open the Crash Rule Script (“CrashRule_IIS.vbs”) in notepad and see for yourself.

Now with DebugDiag you loose the functionality like we had in AD+ to run it from command line. Do we? Not really! Continue reading…

DebugDiag Host

So how do I know what’s available under the hood?

Go to Command prompt and type

C:\>dbghost /? (Obviously you should try from the installation folderJ)

I can analyze the dumps myself? Oh really?

DebugDiag provides analysis feature which you can use from the DebugDiag UI tab called “Advanced Analysis”. By default, right now we have scripts available for analyzing “Crash/Hang Analyzers” and “Memory Pressure Analysis”.

Analysis Scripts are nothing but .ASP pages inside “Scripts” folder which uses somewhat ASP kind of scripting style and uses VBScript to iterate through the structures inside the dumps (which is nicely exposed using an object model) and try to find out known issues or easily identifiable issues so that for simple issues troubleshooting can be done by yourself without calling MS PSS.

More to come which includes Script customization, specific steps to be taken for scenarios like Hang/Crash/Memory related issues etc...

Posted Friday, June 02, 2006 5:10 PM by sukesh | 2 comment(s)

ABC's of Appcmd (command line administration in IIS7)

So what is Appcmd.exe?

This is "one" command line tool to administer IIS7.  In IIS6 several of admin task were done using several scattered VBS script files. This made it difficult to find out what script needs to be run for eg. to get list of worker processes.

So IIS7 is powered with Appcmd.exe which provides all the options you need to administer IIS7.

Following are the options/categories available from a high level

Lets see how we can use it with an example

When I installed LH Server Beta (I rebuild my box quite frequently) I wanted to see how it's like to have 1000 websites running on IIS7.

So I created 1000 websites on my box.

Good scenario to use Appcmd.exe and also my MS-DOS experience. No I'm not gone nutts to create it using the UI :)

Steps required

1. Wanted separate folders for each website
2. Wanted to use same IP address and port for all websites
3. Type a command and leave the box to create all the websites

Keep in mind I'm not talking about Server but my desktop machine. Yea its got 2GB RAM though.

Steps below

1. Created a folder - E:\Websites
2. I wrote a batch file (createsite.cmd) with the following
   MD E:\Websites\Site%1
   appcmd add site /name:"Site%1" /id:%1 /bindings:http/:*:80:site%1 /physicalPath:"E:\Websites\Site%1"
   appcmd start site "Site%1"
3. Now the command to trigger the batch file where my MS-DOS experience came handy
   C:\FOR /L %i IN (2,1,5) DO createsite.cmd %i

FOR command is a batch file loop which simply works like 'for' loop in your favorite language

FOR /L %i IN (2,1,1000) DO createsite.cmd %i

is equivalent to the following in C

for ( i=2; i <= 1000; i++ ) 
   createsite( i );

I started value of 'i' from 2 because "Default Website" has Site ID 1.

Hit enter and wait till the folders and websites are created for each iteration.

Bingo !!! 1000 websites ready to be administered or tested.

So the result would be
E:\Websites folder would have folders called Site2, Site3 etc... and in IIS there would be sites with name Site2, Site3 etc...

Lets revisit the appcmd command above once again

appcmd add site
/name:"Site%1" // website name
/id:%1 // Site ID
/bindings:http/:*:80:site%1 // site would have "All Unassigned" including host header with the site name
/physicalPath:"E:\Websites\Site%1" // Pointing to the physical folder for that site

appcmd start site "Site%1"  // pretty straight forward, it starts the website

What else can I do to extend this scenario?

• Create Application Pools separately for each website
• Create a simple ASP page and drop it in every folder created
• Use TinyGET utility (available with IIS6 Resource Kit) to simulate request

Some other useful command options

Create Backup
C:\>appcmd add backup "backup before screwup"
BACKUP object "backup before screwup" added

List Backup
C:\>appcmd list backup
BACKUP "backup before screwup"

Restore from Backup
C:\>appcmd restore backup "backup before screwup"
Restored configuration from backup "backup before screwup"

Currently Executing Requests
C:\>appcmd list  request
REQUEST "fa00000080000487" (url:GET /highcpu.asp, time:1903 msec, client:localhost)

Will add more of this later...

Posted Friday, June 02, 2006 4:42 PM by sukesh | 2 comment(s)

Evolution of HTTP on Windows Platform

Before the boom of web and internet, HTTP was not so common is everyday life. When internet became more and more widely used, HTTP also grew in usage.

Basic web server serves static HTML type contents and then was the era in which there came several ways to extend the web server and its features. Because of the popularity of the web, on internet most of the companies connected to the web have port 80 (which is the default HTTP port) open on firewalls. Which simply means that to use HTTP most of the companies didn’t have restrictions and thereby became the default and uninterrupted way of global communication?

HTTP on Windows platform was initially used only by IIS for serving web sites. Gradually because of the simplicity and availability (of open port 80) of HTTP several other applications started using HTTP as a medium of transport. Then came the wave of all HTTP based protocols and having another protocol tunnel through HTTP.

RPC over HTTP (I would call it as “RPC through HTTP” rather) for Exchange and SQL to name a few. SQL started supporting HTTP so that you can expose a database over the web using HTTP and SQL queries can be executed using query string and the results returned as XML. Once, more and more application platforms started using HTTP, it became important that we have a uniform & consistent way on Windows platform to use HTTP protocol. So the kernel mode driver called HTTP.SYS was born. Since it’s a kernel mode driver, it gives us a lot of advantages (mainly on 64bit machines with a huge amount of memory for kernel mode caching and getting higher performance) and creates a layer which gives a consistent way of utilizing HTTP protocol. HTTP.SYS also provides API’s if we need to use HTTP protocol directly and bypass IIS. This route would obviously lack other features provided by IIS like health monitoring, easy management etc… but can be utilized for specific needs.

Next big leap – IIS7

Before we look at some of those upcoming features in IIS7, let us see what we lack till the current version of IIS i.e. IIS6.

·         Delegation of administration (very important for website hosting and also for hosting department websites with its own admin)

·         Extensibility (A thorough understanding of IIS is required in order to write high performance ISAPI Filters or Extensions)

·         You can add features but not replace features

·         Managing IIS was not possible when it comes to higher number of websites simply because the MMC style of interface is not efficient.

·         Even if you use ASP.NET (handlers or modules) it works at a lower level than ISAPI Filters/Extensions.

·         Debugging or tracing (A fair amount of tracing features were introduced in IIS6 and more with SP1 but still we resort to post-mortem dumps for 60% of our cases)

·         XCopy deployment of websites.

Now “few” of the highlights from IIS7 features list

·         Delegation of administration to granular level (like attribute level) and users can be either from Active Directory / SAM (workgroup) or custom database

·         You have managed and unmanaged API's to access IIS or for admin purposes (e.g. showing a list of pages currently running and how much CPU each of them are taking). This is possible in IIS6 with SP1 (Windows 2003 SP1) but with extra tools, but in IIS7 it’s available inside the MMC itself.

·         You can search or book mark a website when you have too many sites, for easy administration.

·         Now with the new Integrated Pipeline, you can write managed or unmanaged ISAPI

·         A lot of tracing functionalities are included e.g. Failed Request Tracing etc... Even specifics like “only requests failed with authentication” etc...

·         Since website related details (like default page etc...) are stored inside web.config (not necessarily ASP.NET application) you can deploy sites using XCopy and it simply works.

·         You can enable admin service with a different secure port which can be used for administering IIS remotely (http). This is not an admin website like before but a separate service.

·         You can replace existing IIS components by replacing the dll name in the application.config file. Example custom logging, custom authentication. This simply means that everything is pluggable and only required features are installed or loaded into the process. For example you don't want "Basic Authentication" you don't even install it.  This would reduce attack surface and enables better performance also.

·         My favorite feature, you can enable forms authentication with a single page (uses .NET 2.0) for htm / asp / asp.net / php / cgi / images or any application web site.

You can see some managed code snippets of IIS7 Admin API’s from the following link

http://blogs.msdn.com/carlosag/archive/2006/04/17/MicrosoftWebAdministration.aspx

Make sure you see the piece of code titled “Getting list of executing requests” which is my favorite.

Posted Friday, June 02, 2006 3:58 PM by sukesh | with no comments