Blog of "Brian Murphy-Booth" a.k.a. "Brian Booth"

An error that most IIS 6.0 administrators have probably encountered is "Event ID: 1009" which usually leads to a "503 Service Unavailable" error being displayed in a browser. "503" usually indicates the Application Pool has been disabled for some reason. The IIS support team frequently gets support calls to help resolve this issue and over the years I have compiled a list of steps I use to troubleshoot this. If the following information seems too confusing please let me know and I'll clarify any confusing points. The follow list is broken down into different sections for the various "exit codes" that are in the Event 1009.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

*** Problem Description ***
Many times when an IIS6 application pool terminates the following will be logged.

  1. The Name of the App Pool
  2. The PID
  3. The exit code.

The exit code is the most useful part of the event entry.

  Event Type: Warning
  Event Source: W3SVC
  Event Category: None
  Event ID: 1009
  Date:  1/29/2004
  Time:  10:01:14 AM
  User:  N/A
  Computer: COMPUTERNAME
  Description:
  A process serving application pool 'DefaultAppPool' terminated unexpectedly. The
process id was '3908'. The process exit code was '0x80'.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

*** Resolution ***

Constants: 

// the WAS killed the worker process
#define KILLED_WORKER_PROCESS_EXIT_CODE 0xFFFFFFFD

// the worker process exited ok
#define CLEAN_WORKER_PROCESS_EXIT_CODE  0xFFFFFFFE

// the worker process exited due to a fatal error
#define ERROR_WORKER_PROCESS_EXIT_CODE  0xFFFFFFFF

-----------------------------------------------------------
Concepts that apply to exit codes of both 0x80 and 0xffffffff
  · Make sure "Network Service" and "IWAM_MACHINE" are members of IIS_WPG
  · If a custom identity is being used for the app pool ensure it is in IIS_WPG
  · Make sure that IIS_WPG is included somehow in these User Rights assignments
(example: Everyone group includes IIS_WPG so that is sufficient)
      a. Access this computer from network
      b. Log on as batch job
      c. Bypass traverse checking
  · Ensure that IIS_WPG or members of that group are not in any of the
corresponding "Deny" User Rights
  · Ensure that "NT AUTHORITY\Authenticated Users" and "NT AUTHORITY\Interactive"
are part of the "Users" group.
  · Use FileMon.exe and RegMon.exe to identify ACC DENIED's.
  · If customer has more than ~60 app pools with unique identities, set the
following key:
      HKLM\System\CurrentControlSet\Services\W3SVC\Parameters\UseSharedWPDesktop
(REG_DWORD with value of 1)
-----------------------------------------------------------
The process exit code was '0x80'.
  Means the W3WP.exe never started at all which could be User-Rights related or
NTFS permissions.
  If it is NTFS permissions, that means the AppPool identity doesn't have read
permission to the w3wp.exe file.
  - Use FileMon to troubleshoot
  If it is User Rights related then the AppPool identity failed to logon.
  - Check the 3 User Rights listed above.
-----------------------------------------------------------
The process exit code was '0xffffffff'.
  Means the W3WP.exe process partially started but could not load a dependancy for
some reason.
  This is either permissions/security related or due to mismatched DLL's.

  Try running the AppPool as "Local System"
  - If this works it is a permissions problem for the AppPool Identity. Check the
following "Scenarios" section then follow the "concepts" section above
  - If this fails using System it is a mismatch or missing DLL problem. Check
scenario #1 below then follow the "Loader Snaps" section

Scenarios for 0xffffffff

  1. The first thing that should be checked is whether SP1 is installed
    - If it is, verify that "c:\windows\System32\instsrv\w3core.dll" is either the
RTM version or SP1 version.
    - If it is the RTM version then reapplying SP1 should fix the problem.
    - This issue is said to be fixed for SP2

  2. Is this IIS server a DC and have you reinstalled IIS on another DC?
    I have had two cases where there was a permissions failure reading nodes in the
metabase
    When installing IIS it creates an IIS_WPG group with a somewhat random SID.
    Permissions in the metabase are then set using this unique version of
IIS_WPG.
    When removing IIS on a DC it will delete the IIS_WPG group, if Win2k3 SP1 is
not installed, which is used by all the other DC's running IIS
    When then adding IIS back on to this DC, a new IIS_WPG group is created that
has a new SID
    The pre-existing permissions (older SID) in the other metabases will have
"Unknown User" for the permissions and spawning W3WP.exe under anything other than
SYSTEM will fail with 0xffffffff
    - I found this by enabling Tracing then searching through source code.
    - If you get a Debug Trace using DbgView.exe look for a line that says:
        w3core!W3_SERVER::Initialize [\w3server.cxx @ 526]:Error reading
UseDigestSSP property.  hr = 80070005
    - However It would be easy enough to skip Debug Tracing and just look at the
following nodes using Metabase Explorer.
    - IIS_WPG needs permissions to the following nodes:
      1. MachineName - Read
      2. w3svc/1/Filters (or any other filters node) - Read/Write
      3. w3svc/AppPools - Special (Query Unsecure Property)

  3. Has the customer modified default DCOM security?
    - We have run into an issue where the customer had modified
the default Launch and Activation Permissions in Component Services.
    - The customer removed Local Launch and Local Activation for the Everyone
group.
    - Here is the section of loader snaps output that is a hint of this scenario:

LDR: LdrGetProcedureAddress by NAME - CoMarshalInterface
LDR: LdrGetProcedureAddress by NAME - CoUnmarshalInterface
LDR: LdrGetProcedureAddress by NAME - CoReleaseMarshalData
(15c8.914): Unknown exception - code 80070005 (first chance)
LDR: UNINIT LIST
          (1) [iisres.dll] c:\windows\system32\inetsrv\iisres.dll (0) deinit 0
LDR: Unmapping [iisres.dll]
LDR: Derefcount IISMAP.dll (0)

-----------------------------------------------------------
The process exit code was '0xc0000005'.
  This is a crash.
  Troubleshoot using a Debugger. (Debug Diagnostics)
-----------------------------------------------------------
The process exit code was '0xff'.
  Process shut down "gracefully" for some reason.
  Troubleshoot as a crash and see who called TerminateProcess or ExitProcess.
-----------------------------------------------------------
The process exit code was '0x0'.
  This would be typical if you had w3wp.exe configured to launch under a debugger
and you never did a "Go" in the debugger windows.
  Launch gflags.exe and clear the debugger setting for w3wp.exe
-----------------------------------------------------------
Loader Snaps Section - These steps are not for getting memory dumps. This explains
how to easily get the reason that a module (DLL) failed to load which is one reason for a 0xffffffff.

  1. Send customer "gflags.exe" (can be obtained from Debugging Tools for
Windows)
  2. Double-click gflags.exe
  3. Go to the "Image File" tab
  4. Enter "w3wp.exe" for the image then press "tab"
  5. Put a check in "Show loader snaps"
  6. Put a check in "Debugger" and enter
        "NTSD.exe -logo c:\temp\LDR.log -g -G -r 0" (<-- that is a zero)
  7. Ensure the folder from the previous step has adequate read/write permissions
for the Identity that is launching the AppPool.
  8. Ensure that the AppPool is enabled
  9. Reproduce the problem. - At this point w3wp.exe should spawn under NTSD.exe,
write to the log, then shut down.
  10. Have customer send the LoaderSnaps.log output found in the c:\temp folder.
  11. Search for the text of "exception" or "failed" (you'll probably find what you
want near the end of the log)
  12. Lookup the listed error number using err.exe or hrplus.exe. The DLL that it
is having trouble on is the line just previous to the "exception/failed" line.
  13. Take the logical steps to address whatever the error is describing.

  Notes - Don't forget to reverse these settings when done.
            - If you identify a DLL that is the wrong version usually simply
reapplying the relevant hotfix or service pack will resolve the issue.
 

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Where is the tool?

  http://www.iis.net/downloads/default.aspx?tabid=34&g=6&i=1434 

What is this tool?

  This is an ASP.NET application that I wrote a couple years ago that is meant to be called from Internet Explorer on an actual client machine. The tool (aspx page) attempts to look at all the common settings that contribute towards successful Kerberos authentication and delegation. I had originally written a  simple ASP page for my co-workers to demonstrate how to see whether a request had authenticated with Kerberos or NTLM by doing Response.Write Request.ServerVariables("HTTP_AUTHORIZATION"). After a little while I decided "why stop there?!" The tool will now check all the common pitfalls of Kerberos authentication except for a couple issues that I can't check for when the web request fails entirely (if my page won't even run, how can I check those settings?).

I'd like to hear your feedback!! If you have problems, constructive criticism, or praise please let me know. If you have comments or suggestions that will benefit the masses, I'll try to find time to implement your suggestions into the tool.

Usage Tips:

• READ THE "Explanation" SECTIONS!! This tool is meant to be a teacher. If you don't read your material, you probably aren't going to understand what to do or at least why you're doing something that it suggests you do.
• Did I mention that you should read the "Explanation" sections?? Do it!!
• There are some "Fix This" buttons that help automate addressing issues. Although it isn't 100% clear in some of the output, it is probably easiest to use the "Fix This" buttons instead of manually following any KB articles that are also included in the output. You can, however, do whichever you prefer since either method should fix the applicable configuration problem.
• If the DelegConfig tool doesn't work as expected, please see the "KNOWN PROBLEMS AND WORKAROUNDS" section below
• If you end up making changes to your configuration based on what the tool reports, it is common that you'll need to log on/off the workstation machine and/or restart the IIS service or back-end service before you see Kerberos/Delegation start to work as desired.
• Part of "Delegation" is authenticating with a back-end service. Don't forget to add the back-end server information with the applicable hyperlink to make sure everything looks good there too.

---------------------------------------------------------------
REQUIRED COMPONENTS:
---------------------------------------------------------------
  1. All files in this ZIP are required for the ASPX page to operate properly.
  2. ASP.NET version 1.1.4322.2300 or higher (also works with v2.0.50727) must be installed on the IIS server.

---------------------------------------------------------------
OPTIONAL COMPONENTS:
---------------------------------------------------------------
  1. For more accurate results the .Net framework should be installed on the Workstation/Client

---------------------------------------------------------------
SETUP:
---------------------------------------------------------------
  1. Unzip files to desired location that is *local* to the IIS server.
  2. In the IIS MMC create a new virtual directory that points to the folder with the unzipped files.
  3. Configure the virtual directory as an IIS application
    a. In the IIS MMC right-click the vDir and choose Properties.
    b. On the "Virtual Directory" tab click "Create" (if already configured as an IIS app you'll see a "Remove" button instead)
  4. Ensure that "Scripts Only" (recommended) or "Scripts and Executables" is enabled for this vDir.

---------------------------------------------------------------
USAGE:
---------------------------------------------------------------
  1. View the Default.aspx page through a web browser with an address such as "http://MyServer/MyVirtualDirectory/"

---------------------------------------------------------------
KNOWN PROBLEMS AND WORKAROUNDS.
---------------------------------------------------------------
General:
  Generally speaking if there are any problems running this tool's Default.aspx page (errors of any sort) it is probably because Kerberos isn't working yet. :-p. In that case it is best to start by requesting the page locally from the IIS server. Certain types of problems only exist when IE attempts to connect to IIS using Kerberos and Kerberos is usually not used when local to the IIS server. Requesting the Default.aspx locally from the server will avoid many types of problems this DelegConfig tool can encounter. Once you can get this page working locally from the IIS server the tool will then report some of the problems that could affect Kerberos and/or Delegation. Once some of those items are addressed and Kerberos is closer to working you may be able to do additional checks by requesting the ASPX page from a remote IE machine.

--------------
Problem:
[HttpException (0x8007052e): Failed to start monitoring changes to '\\ServerName\ShareName'.]

  ASP.net uses a hierarchical system of reading configuration files starting with Machine.Config, then web.config from the root of the web site, then finally web.config file from the virtual directory the ASPX application is running. Any time one of these files changes, IIS wants to be able to reload the file and restart the web application to ensure the most recent configuration is being used. If the dot net framework is not able to read from the root of the web site using the ASPNET account, this error will be returned. This is typical of when the home directory is a UNC path.

Workaround:
  1. If running IIS 6.0 this can sometimes be fixed by disabling "Run www service in IIS 5.0 Isolation Mode".
  OR
  2. To ensure that ASPNET can read from the root of the web site, temporarily change the "Home Directory" of the site to a local path. Once you have resolved the Kerberos and Delegation issues based on the results of the ASPX application you can change the home directory back to the desired UNC path.

--------------
Problem:
  Continuous password prompt with underlying 401.1 response. There are 2 common possibilities for this.

Additional Information:
  Looking in the security event logs shows:

POSSIBILITY ONE (notice the Logon Process of Kerberos):
  Event Type: Failure Audit
  Event Source: Security
  Event Category: Logon/Logoff
  Event ID: 529
  Date:  1/1/2005
  Time:  6:00:00 PM
  User:  NT AUTHORITY\SYSTEM
  Computer: COMPUTERNAME
  Description:
  Logon Failure:
    Reason:  Unknown user name or bad password
    User Name: 
    Domain:  
    Logon Type: 3
     Logon Process: Kerberos
    Authentication Package: Kerberos

Workaround:
  You receive an "HTTP Error 401.1 - Unauthorized: Access is denied due to invalid credentials" error message when you try to access a Web site that is part of an IIS 6.0 application pool
  http://support.microsoft.com/?id=871179

POSSIBILITY TWO (notice the strange Logon Process):
  Event Type: Failure Audit
  Event Source: Security
  Event Category: Logon/Logoff
  Event ID: 537
  Date:  1/1/2005
  Time:  6:00:00 PM
  User:  NT AUTHORITY\SYSTEM
  Computer: COMPUTERNAME
  Description:
  Logon Failure:
    Reason:  An error occurred during logon
    User Name: someuser
    Domain:  SOMEDOMAIN
    Logon Type: 3
    Logon Process: Ðù²
    Authentication Package: NTLM
    Workstation Name: COMPUTERNAME
    Status code: 0xC000006D
    Substatus code: 0x0
    Source Network Address: 127.0.0.1

Workaround:
  You receive error 401.1 when you browse a Web site that uses Integrated Authentication and is hosted on IIS 5.1 or IIS 6
  http://support.microsoft.com/?id=896861

Note: All of your Kerberos configuration questions can be answered by using the DelegConfig tool that I wrote. You can find that tool here 

Literally 99% of all Kerberos problems revolve around an incorrect, missing, or duplicate ServicePrincipalName (SPN).  To be honest, the concept of an SPN is so simple that I am often confused that other people don't understand even after I explain. I suppose it is the 5+ years that I've had of helping people configure and troubleshoot Kerberos related issues that have finally made it all clear to me ;-p. I like to think in simple terms instead of making things complex . This is a carry-over from my Algebra-1 days when my teacher used to pick the easiest problem possible when explaining a concept.

Think of an SPN as a “username” used to identify a program that is busy dealing with credentials. And we're only allowed to talk to this program using its “username”. PERIOD. Simple! Yes, that's all an SPN is: a "username". And as with any username, the name itself isn't really that important. It is merely to make identifying a person (or entity) easier to remember to humans. In this particular case, however, there are some naming conventions for this "username". Okay, so what username (SPN) is the right one? And where do we set it? These 2 questions are where all the confusion lies. We split the SPN into 2 parts and occasionally 3 parts: The first part is the “service type” and the second part is the “host name”. And sometimes the 3rd part is present which is the “port”. In the end, however, all these different parts are simply used to come up with this "username" that we call the ServicePrincipalName.

Let’s say I wanted to connect to a process called BrianService.exe. And the DNS name to route my connection was blah.overthere.com. As the designer of this weird service I might come up with a “service type” of BRIAN. So the SPN would be BRIAN/blah.overthere.com. Okay, and where do we set that? Simple, simple, simple. If my BrianService.exe process is running under “DOMAINNAME\someAccount” then we’d set the BRIAN/blah.overthere.com SPN on “DOMAINNAME\someAccount”. If my process (BrianService.exe) were running as something like “Network Service”, “Local Service”, or “Local System” then I’d set BRIAN/blah.overthere.com on the computer account itself that is running that process. If you ever change the account that is running your program then you need to remove the SPN from the original account and set it on the new account because we can't have the same username assigned to multiple "people" (or accounts in our case).

 Recap. An SPN is just a *name* that we've given to a "service" which is in the format of ServiceType/HostName and occasionally ServiceType/HostName:PortNumber. And it is set on which ever account is handling authentication for that service. I should also note that you as an administrator don’t get to pick whether you use a port. I used to think that maybe I could throw a port number on an SPN if I wanted to make it more secure. But it is the client application that has the decision built in on whether to use a port.

Okay, so let’s make this more complicated by using more realistic names. But while I do that I want you to maintain faith in what I just explained above regarding how simple these concepts are. Let’s say you’re connecting to an IIS server with a machine name of “iis-prod-01”. And let’s say the active directory domain name is “company.com.” In Internet Explorer you use an address of http://someInventedName. The “application pool” (i.e. the w3wp.exe process) is running under the account of “COMPANY\myserviceAccount”. With the knowledge that the web service’s Kerberos “service type” is “HTTP” (don’t confuse this with the browser’s protocol type) you’re probably thinking we can set an SPN of “HTTP/someInventedName” on “COMPANY\myserviceAccount”.  Doh!! Sorry no. Almost, but Kerberos would probably not work with that. The problem with that idea is that you have to know how name resolution is working also because it is ultimately name resolution that dictates what the "host name" part of the SPN should be. If you open a CMD prompt and ping someInventedName, it will most likely resolve to someInventedName.company.com. Therefore the SPN that “IE” will request is “HTTP/someInventedName.company.com”. IE was not programmed to request an SPN using the port so that part of the SPN is not needed nor can it ever be used. What if the ping did show the name as just “someInventedName”? Then IE would in-fact use Kerberos with an SPN of “HTTP/someInventedName” When dealing with NetBIOS names, because name resolution can be affected by many things, the key is to make sure an SPN of both “HTTP/someInventedName” and “HTTP/someInventedName.company.com” are set on the “COMPANY\myserviceAccount” account. Or the way I prefer to say that is you need to create an SPN that represents both the NetBIOS name and the Fully Qualified name.

Okay, so I can hear what many of you are thinking. “But I thought that KB article said to set the SPN on the computer account!” Well, yes, that would be accurate *IF* the process handling authentication was running as “SYSTEM”. If the process for that service is not running as SYSTEM (or Network Service, or Local Service) then you can’t set the SPN on the computer account (well you can but Kerberos isn’t going to work).

Recap 2: An SPN should actually be in the format of ServiceType/NetBIOSName *and* ServiceType/FQDN. And we *always* set that on whatever account is running the process that is handling the authentication. Read the above paragraphs a couple times and just maintain faith that it is really that simple. Don’t complicate it with questions!!  

I want to mention one last thing before I go. Whenever a computer is joined to a domain, it is assigned 2 SPN's by default: HOST/netbiosName, and HOST/FQDN.com. netbiosName being the machine name of the computer you're joining to the domain, and FQDN.com being the fully qualified machine name. These two SPN's use the generic "HOST" service type which includes all the various services that *come* with Windows. Therefore, if you connect to http://machineName or http://machineName.company.com, you will already have SPN's set that will handle Kerberos when using those names. Or if you connected to \\machineName\SomeShareName you'd also be all set for Kerberos (UNC's need a "CIFS" SPN which is included under "HOST" also). For a full list of the different service types included in HOST please see Table 1 of this technet article.

--Brian Murphy-Booth

Note: Although you can get Visual Studio .NET 2003 to work properly on Vista, Microsoft does not support this scenario. "Not supported" doesn't mean "won't work". It just means if you run into any trouble you just can't call the MS support line for assistance with this issue. Microsoft does, however, fully support *running* a .NET v1.1 app on Vista.

Because I support both IIS and ASP.NET here at Microsoft, I find it necessary to have both "Visual Studio .NET 2003" and "Visual Studio 2005" installed. I need to be able effectively troubleshoot issues for whatever product our customers need help with. But like any other tech junky, I like to have the latest-and-greatest OS installed so that I can learn the various features of the OS through day-to-day use. For this reason, I have Windows Vista installed on my primary machine. But, particularly when I was less familiar with the IIS 7.0 UI, getting ASP.NET 1.1 to run on IIS 7.0 was frustrating. Here are some of the errors I encountered while trying to get this setup:

---

Microsoft Development Environment
Error while trying to run project: Unable to start debugging on the web server. Could not start ASP.NET or ATL Server debugging.
Verify that ASP.NET or ATL Server is correctly installed on the server.

---

Microsoft Developement Environment
Error while trying to run project: Unable to start debugging on the web server. You do not have permissions to debug the server.
Verify that you are a member of the "Debugger Users" group on the server.

---

ASP.NET Version Mismatch
Visual Studio .NET has detected that the Web server is running ASP.NET version 1.0. The Web application you are creating or opening can be configured to be compliant with ASP.NET 1.0.
However, the application will not be able to use new features from ASP.NET 1.1.

---

To set this up properly so the above errors don't occur, follow these steps.

1. Ensure that .NET 1.1 --->SP1<--- or higher is properly installed. Vista does not include .NET v1.1 by default. Because .NET 1.1 is not included by default, .NET v1.1 *SP1* is also not included. Without SP1, W3WP.exe will crash when running an appPool under v1.1 due to DEP To check this, make sure that c:\Windows\Microsoft.NET\Framework\v1.1.4322\mscorsvr.dll is version 1.1.4322.2032 or higher. Unless you are 100% sure that SP1 for .NET is installed, you *really* should double-check this.
2. Enable IIS 6.0 compatibility Open Control Panel Double-click Programs and Features Expand Internet Information Services Expand Web Management Tools Check IIS 6 Management Compatibility
3. Register v1.1 with IIS Open a CMD prompt Change your directory to c:\Windows\MIcrosoft.net\Framework\v1.1.4322 Run "aspnet_regiis -ir" "ir" registers v1.1 with IIS but doesn't change any existing script mappings aspnet_regiis should also create a new AppPool under "Application Pools" called "ASP.NET 1.1" that is configured with the "Classic" pipline, and "Enable32BitAppOnWin64" set to true if a 64-bit OS.
4. Make sure IIS permits running ASP.NET. Open the IIS manager.  Highlight your Computer Name. Double-click ISAPI and CGI Restrictions Select ASP.NET v1.1.4322 Click Allow in the Actions section in the upper right.
5. Make the new ASP.NET 1.1 appPool the default. Open the IIS manager Select the Sites folder. Under Actions on the upper right, click Set Web Site Defaults... Change the Application Pool setting to ASP.NET 1.1
6. **ALTERNATIVE step to 5** - Change the AppPool to ASP.NET 1.1 -->after<-- creating the ASP.NET project instead of making it the default. Create the v1.1 ASP.NET project via Visual Studio. Attempting to run the project at this point will fail if the 1.1 appPool is not the default. Open the IIS manager. Right-click the newly create application directory and choose Advanced Settings Change the Application Pool to ASP.NET 1.1 Go back to Visual Studio and attempt to run/debug project.

Happy coding!!