Measuring incoming and outgoing bandwidth per Application Pool

This blog post is about working in a big company where the left hand often times doesn't know what the right hand does.

These days we are talking quite a bit about how to best measure resource consumption (disk, memory, CPU, requests per second, incoming, outgoing bandwidth) of individual Application Pools. As part of this exercise we looked at IIS and HTTP.SYS performance counters and found a pretty embarrassing issue when it comes to measuring bandwidth.

You might know that HTTP.SYS does a lot of content caching for IIS. The benefit is that some requests can be directly returned from kernel mode without ever having to go to user mode where IIS lives. This can speed up response time quite drastically. IIS has a per Web-Site performance counter that measures bandwidth ("Web Service", "Bytes Received/sec" and Web Service", "Bytes Sent/sec"). The problem is that this only measures the bandwidth for requests that make it to user mode. All requests that get cached by HTTP.SYS in kernel-mode will not be measured.

The good thing is that HTTP.SYS has the same performance counters ("HTTP Service Url Groups", "BytesReceivedRate" and "HTTP Service UrlGroups", "BytesSentRate"). Great!

Here comes the issue though: the counters are per-Url Group and not per Web-Site or per Application Pool and there is no obvious way to figure out which Web Site or Application Pool is in which Url Group (what Url Groups are is explained here). HTTP.SYS provides no APIs to get to this information. The only way to make the connection between a Url Group and an Application Pool is to parse the output of the "NETSH.EXE HTTP SHOW SERVICESTATE" command which shows a snapshot of the HTTP.SYS service configuration. By parsing out the UrlGroups and the associated Application Pool you can now write a program that measures the incoming and outgoing bandwidth on a per Application Pool basis.

Last night I wrote a PowerShell script that does exactly that. Here roughly the steps the script follows:

1. Execute the netsh command and store the output in variable $file_lines
2. Loop through each line and find the string "URL group ID: "
3. If the string is found we look for the associated AppPool which is shown after the string "Request queue name: "
4. Ignore non-IIS Url Groups ("Request queue is unnamed.")
5. Add URL Group/AppPool combo to a hash table which has the Application Pool as a key and an array of Url Groups as its value.
6. Create a perf counter for BytesSentRate, BytesReceivedRate and AllRequests.
7. Loop through the Url Groups for each Application Pool and add the values together to get the total value.
8. Write values to console.

$URLGROUP_STRING = "URL group ID: ";
$APPPOOL_NAME_STRING = "Request queue name: ";
$NON_IIS_URLGROUP = "Request queue is unnamed.";
$UrlPoolHashTable = @{};


$file_lines = &"c:\windows\system32\netsh.exe" http show servicestate

for($i = 0; $i -lt $file_lines.Length; $i++)
{
    $res = $file_lines[$i].IndexOf("$URLGROUP_STRING");    
    if ($res -ge 0)
    {
        $urlgroup = $file_lines[$i].SubString($res + $URLGROUP_STRING.Length);        
        while (++$i-lt $file_lines.Length)
        {        
            $res = $file_lines[$i].IndexOf("$APPPOOL_NAME_STRING");
            if ($res -ge 0)
            {
                $appPool = $file_lines[$i].`
                                SubString($res + $APPPOOL_NAME_STRING.Length);
                if ($false -eq $appPool.Contains("$NON_IIS_URLGROUP"))
                {
                    if ($UrlPoolHashTable.ContainsKey($appPool))
                    {
                        $UrlPoolHashTable[$appPool] += "$urlgroup";                    
                    }
                    else
                    {
                        $UrlPoolHashTable[$appPool] = , "$urlgroup";
                    }                    
                }
                break;
            }
        }
    }    
}


foreach ($appPoolkey in $UrlPoolHashTable.Keys)
{
    $bwOut = new-object System.Diagnostics.PerformanceCounter(
                            "Http Service Url Groups","BytesSentRate");
    $bwIn  = new-object System.Diagnostics.PerformanceCounter(
                            "Http Service Url Groups","BytesReceivedRate");
    $req  = new-object System.Diagnostics.PerformanceCounter(
                            "Http Service Url Groups","AllRequests");

    $inTotal = 0;
    $outTotal = 0;
    $reqTotal = 0;
    foreach ($urlgroup in $UrlPoolHashTable[$appPoolkey])
    {
        $bwOut.instanceName = $urlgroup;
        $bwIn.instanceName = $urlgroup;
        $req.instanceName = $urlgroup;
        $outTotal += $bwOut.RawValue;
        $inTotal += $bwIn.RawValue;
        $reqTotal += $req.RawValue;        
    }
    "Application Pool: $appPoolKey";
    "             Bandwidth out: " + $outTotal + " bytes";
    "             Bandwidth in:  " + $inTotal + " bytes";
    "             Request Total: " + $reqTotal;
        
    
}    

P.S: The counter gets reset when the Url Group is removed, e.g. when the W3SVC service is stopped. Also: this probably only works with the english version of Windows. I'm assuming that localized Windows versions will have different strings to search for.