cpattekar

Running IIS7 worker processes under debugger

cpattekar — Tue, 13 May 2008 20:49:00 GMT

This blogs explains how to configure debugger such that worker processes launch under debugger in Windows Server 2008.

Step 1: Install Debugging tools for windows from http://www.microsoft.com/whdc/devtools/debugging/default.mspx. This will install ntsd.exe which is what we are going to use. Note the directory to which the debuggers got installed.

Step 2: In this step, we will create an entry in Windows System Registry for the debugger. Simply open Windows registry editor (regedit.exe) and navigate to:

"HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options".

Step 3: Right click on "Image File Execution Options" node and click on New > Key. Name this key "w3wp.exe".

Step 4: In the right pane of Registry editor, right click and add New > String value. Call this key "Debugger".

Step 5: Double click Debugger and in the "Value data" text field, enter the following string:

<debugger path>\ntsd.exe -server npipe:pipe=w3wp%d -G -g

-g tells the debugger not to breakin on the initial breakpoint when the worker process is starting up.

-G tells the debugger not to breakin on the final breakpoint when the worker process is exiting.

Step 6: Make a request to IIS using internet explorer or your faviourite client so that the worker process starts up. Start a command prompt and CD to the debugger folder. Type "tlist -t" to get a tree view of processes running on the machine. Look for w3wp.exe and it should be running under ntsd.exe like:

    svchost.exe (1628)
      ntsd.exe (1308)
        w3wp.exe (2212)

Step 7: Connect to the debugger via remote.exe using the following command:

<debugger path>\ntsd -remote npipe:server=<machine name>,pipe=w3wp<debugger PID>

where <debugger PID> is the PID of the ntsd.exe process under which the worker process is running. In the above example, it will be 1308. Lot of people confuse this with the PID of the worker process which doesn't work.

Note: It is recommended that you match the bitness of debugger with the bitness of w3wp.exe. I mean that if you are running 32 bit worker process on 64 bit OS (in wow64), you should use 32 bit debuggers. In this case, install 32 bit debuggers on the machine and use the following registry key:

"HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options"

Note: This behavior for wow64 debugging is going to change for next version of windows (win7). I will blog about the new behavior when win7 releases.

Stress testing IIS

cpattekar — Thu, 08 May 2008 23:02:00 GMT

As promised in my first blog, I am writing about stress testing IIS in this blog.

The idea behind IIS stress testing is to bombard the server with multiple simultaneous requests of different kind so that the CPU usage goes to 100%. This helps detecting memory leaks, access violations and deadlocks.

There is a fine line between stress testing and reliability testing. In stress testing, the response of the server for every request doesn't matter as long as we verify that a few of the responses are as expected. As I said the main aim of stress testing is to keep the CPU pegged without too much worrying about the response. In reliability also, we hit the server to keep it busy, but we don't want to hit it as hard. In reliability testing, we validate each and every response coming back from the server to make sure it is as expected.

Talking about stress, while the machine is being bombarded by requests, we make a few simple requests called Stress Verification Tests (SVTs) every 15 minutes and validate them to make sure server is responding as expected. We also run a script in parallel that fetches various performance counters like memory usage, handle count etc and validate them to make sure they are in the acceptable range.

We have setup the server and its multiple physically separate clients on the same rack so that we get enough throughput. To begin with, we create multiple sites, applications and app pools with a variety of settings on the server. Then we start our script to monitor performance counters. We make a single pass of SVTs before starting stress on the server to make sure all configuration is good. We the start the clients that use WCAT (available on iis.net for download). WCAT is web capacity analyzer tool that bombards the server using multiple threads. I will try to blog about WCAT in a separate blog.

We have multiple types of stress runs, some are overnight and some run for 3 weeks. When a run doesn't hit any failure for the duration of run, we consider that passed. Most of the failures get caught if an SVT fails or if there is an AV in the worker process or if there is a performance counter that is outside of expected range.

SVT failures can be quite challenging because we might have overstressed the server and there is no way server can respond to SVT request. These are just like a false alarm. There is not much anyone can do here.

For nightly stress runs, we have 100+ servers and each server has 1 or more clients. Managing all these machines is a big challenge. We have a nice logging mechanism that logs the status of these runs to a database and a web application as a UI. We also log few interesting performance counters to the database and view graphs of those performance counters in the web application that uses silverlight for charting. Worker processes run under debugger with some special debugger extensions that directly logs about failures to the database. Everything is automated!

Why do we need 100+ runs every night? The stress client uses different kinds of requests for different runs. Some runs use CGI requests, some runs use plain HTM requests, some runs use ASP, some runs use ASPNET, some runs use combination of these. It is not necessary that if one run that uses HTM requests fails, all will fail. Different machines have different hardware configuration, so one might fail but others might pass. We also have different settings in IIS config so that may give us different results. E.g. if FREB is enabled, server may get more stressed because worker process is doing more work per request.

Stress testing is overwhelming but I hope the reader of this blog got an idea of what it is and the challenges we face. In the next blog, I will write more about WCAT.

Who is CJ Pattekar from IIS Team?

cpattekar — Thu, 08 May 2008 22:22:00 GMT

Hello,

A year ago I setup my account on IIS.net so that I could write blog and answer questions from IIS customers on forums. But the main focus at that time was to ship Windows Server 2008 and then the focus switched to shipping IIS out of band modules. Although the shipping process doesn't stop and there will always be something that takes higher priority, I feel this is about the time I should start interacting with IIS customers through various channels. The obvious one is www.iis.net. So I am jumping in and writing my first blog here.

My full name is Chittaranjan Pattekar and I am from India. I have shortened my first name to CJ for the convenience of my collegues :). I did my masters in computer science from Syracuse university in NY state. I have been working with the IIS "test" team for last 2+ years.

I own stress, reliability and some internal tools to image our test machines. I also own testing some legacy IIS features like performance counters and CGI, but most of my time is spent in stress testing IIS and maintaining the infrastructure tools.

In my next blog, I will write about IIS Stress testing tools and challenges.

-CJ