Blog of "Brian Murphy-Booth" a.k.a. "Brian Booth" : IIS News Item

How to troubleshoot an IIS "Event ID: 1009" error.

brian-murphy-booth — Thu, 22 Mar 2007 16:20:00 GMT

An error that most IIS 6.0 administrators have probably encountered is "Event ID: 1009" which usually leads to a "503 Service Unavailable" error being displayed in a browser. "503" usually indicates the Application Pool has been disabled for some reason. The IIS support team frequently gets support calls to help resolve this issue and over the years I have compiled a list of steps I use to troubleshoot this. If the following information seems too confusing please let me know and I'll clarify any confusing points. The follow list is broken down into different sections for the various "exit codes" that are in the Event 1009.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

*** Problem Description ***
Many times when an IIS6 application pool terminates the following will be logged.

1. The Name of the App Pool
2. The PID
3. The exit code.

The exit code is the most useful part of the event entry.

Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1009
Date:  1/29/2004
Time:  10:01:14 AM
User:  N/A
Computer: COMPUTERNAME
Description:
A process serving application pool 'DefaultAppPool' terminated unexpectedly. The
process id was '3908'. The process exit code was '0x80'.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

*** Resolution ***

Constants:

// the WAS killed the worker process
#define KILLED_WORKER_PROCESS_EXIT_CODE 0xFFFFFFFD

// the worker process exited ok
#define CLEAN_WORKER_PROCESS_EXIT_CODE 0xFFFFFFFE

// the worker process exited due to a fatal error
#define ERROR_WORKER_PROCESS_EXIT_CODE 0xFFFFFFFF

-----------------------------------------------------------
Concepts that apply to exit codes of both 0x80 and 0xffffffff
· Make sure "Network Service" and "IWAM_MACHINE" are members of IIS_WPG
· If a custom identity is being used for the app pool ensure it is in IIS_WPG
· Make sure that IIS_WPG is included somehow in these User Rights assignments
(example: Everyone group includes IIS_WPG so that is sufficient)
      a. Access this computer from network
      b. Log on as batch job
      c. Bypass traverse checking
· Ensure that IIS_WPG or members of that group are not in any of the
corresponding "Deny" User Rights
· Ensure that "NT AUTHORITY\Authenticated Users" and "NT AUTHORITY\Interactive"
are part of the "Users" group.
· Use FileMon.exe and RegMon.exe to identify ACC DENIED's.
· If customer has more than ~60 app pools with unique identities, set the
following key:
      HKLM\System\CurrentControlSet\Services\W3SVC\Parameters\UseSharedWPDesktop
(REG_DWORD with value of 1)
-----------------------------------------------------------
The process exit code was '0x80'.
In certain cases when w3wp.exe goes away IIS will attempt to determine the exit code by calling the "GetExitCodeProcess" API.
If the reason cannot be determined by this API then the code returned is 0x80.
"0x80" means "ERROR_WAIT_NO_CHILDREN" which means "There are no child processes to wait for"

This typically means the W3WP.exe never started at all which could be User-Rights related or
NTFS permissions.
If it is NTFS permissions, that means the AppPool identity doesn't have read
permission to the w3wp.exe file and/or supporting DLLs
- Use FileMon to troubleshoot
If it is User Rights related then the AppPool identity failed to logon.
- Check the 3 User Rights listed above.
-----------------------------------------------------------
The process exit code was '0xffffffff'.
Means the W3WP.exe process partially started but could not load a dependancy for
some reason.
This is either permissions/security related or due to mismatched DLL's.
Try running the AppPool as "Local System"
- If this works it is a permissions problem for the AppPool Identity. Check the
following "Scenarios" section then follow the "concepts" section above
- If this fails using System it is a mismatch or missing DLL problem. Check
scenario #1 below then follow the "Loader Snaps" section

Scenarios for 0xffffffff

1. The first thing that should be checked is whether a Windows Service pack is installed
- If it is, verify that "c:\windows\System32\instsrv\w3core.dll" is either the
RTM version or SP1 version.
- If it is the RTM version then reapplying the service pack should fix the problem.

2. Is this IIS server a DC and have you reinstalled IIS on another DC?
    I have had two cases where there was a permissions failure reading nodes in the
metabase
    When installing IIS it creates an IIS_WPG group with a somewhat random SID.
    Permissions in the metabase are then set using this unique version of
IIS_WPG.
    When removing IIS on a DC it will delete the IIS_WPG group, if Win2k3 SP1 is
not installed, which is used by all the other DC's running IIS
    When then adding IIS back on to this DC, a new IIS_WPG group is created that
has a new SID
    The pre-existing permissions (older SID) in the other metabases will have
"Unknown User" for the permissions and spawning W3WP.exe under anything other than
SYSTEM will fail with 0xffffffff
    - I found this by enabling Tracing then searching through source code.
    - If you get a Debug Trace using DbgView.exe look for a line that says:
        w3core!W3_SERVER::Initialize [\w3server.cxx @ 526]:Error reading
UseDigestSSP property. hr = 80070005
    - However It would be easy enough to skip Debug Tracing and just look at the
following nodes using Metabase Explorer.
    - IIS_WPG needs permissions to the following nodes:
      1. MachineName - Read
      2. w3svc/1/Filters (or any other filters node) - Read/Write
      3. w3svc/AppPools - Special (Query Unsecure Property)

3. Has the customer modified default DCOM security?
    - We have run into an issue where the customer had modified
the default Launch and Activation Permissions in Component Services.
    - The customer removed Local Launch and Local Activation for the Everyone
group.
    - Here is the section of loader snaps output that is a hint of this scenario:

LDR: LdrGetProcedureAddress by NAME - CoMarshalInterface
LDR: LdrGetProcedureAddress by NAME - CoUnmarshalInterface
LDR: LdrGetProcedureAddress by NAME - CoReleaseMarshalData
(15c8.914): Unknown exception - code 80070005 (first chance)
LDR: UNINIT LIST
(1) [iisres.dll] c:\windows\system32\inetsrv\iisres.dll (0) deinit 0
LDR: Unmapping [iisres.dll]
LDR: Derefcount IISMAP.dll (0)

-----------------------------------------------------------
The process exit code was '0xc0000005'.
This is a crash.
Troubleshoot using a Debugger. (Debug Diagnostics)
-----------------------------------------------------------
The process exit code was '0xff'.
Process shut down "gracefully" for some reason.
Troubleshoot as a crash and see who called TerminateProcess or ExitProcess.
-----------------------------------------------------------
The process exit code was '0x0'.
This would be typical if you had w3wp.exe configured to launch under a debugger
and you never did a "Go" in the debugger windows.
Launch gflags.exe and clear the debugger setting for w3wp.exe
-----------------------------------------------------------
Loader Snaps Section - These steps are not for getting memory dumps. This explains
how to easily get the reason that a module (DLL) failed to load which is one reason for a 0xffffffff.

1. Send customer "gflags.exe" (can be obtained from Debugging Tools for
Windows)
2. Double-click gflags.exe
3. Go to the "Image File" tab
4. Enter "w3wp.exe" for the image then press "tab"
5. Put a check in "Show loader snaps"
6. Put a check in "Debugger" and enter
"NTSD.exe -logo c:\temp\LDR.log -g -G -r 0" (<-- that is a zero)
7. Ensure the folder from the previous step has adequate read/write permissions
for the Identity that is launching the AppPool.
8. Ensure that the AppPool is enabled
9. Reproduce the problem. - At this point w3wp.exe should spawn under NTSD.exe,
write to the log, then shut down.
10. Have customer send the LoaderSnaps.log output found in the c:\temp folder.
11. Search for the text of "exception" or "failed" (you'll probably find what you
want near the end of the log)
12. Lookup the listed error number using err.exe or hrplus.exe. The DLL that it
is having trouble on is the line just previous to the "exception/failed" line.
13. Take the logical steps to address whatever the error is describing.

Notes - Don't forget to reverse these settings when done.
- If you identify a DLL that is the wrong version usually simply
reapplying the relevant hotfix or service pack will resolve the issue.

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

DelegConfig (Kerberos/Delegation Configuration Reporting Tool)

brian-murphy-booth — Fri, 09 Mar 2007 19:10:00 GMT

Where is the tool?

For 2.0+ framework - IIS 6.0, 7.0, 7.5 - http://blogs.iis.net/brian-murphy-booth/archive/2009/04/22/delegconfig-v2-beta.aspx

For 1.1 framework - IIS 5.0, IIS 6.0 - http://www.iis.net/downloads/default.aspx?tabid=34&g=6&i=1434

What is this tool?

This is an ASP.NET application that I wrote a couple years ago that is meant to be called from Internet Explorer on an actual client machine. The tool (aspx page) attempts to look at all the common settings that contribute towards successful Kerberos authentication and delegation. I had originally written a simple ASP page for my co-workers to demonstrate how to see whether a request had authenticated with Kerberos or NTLM by doing Response.Write Request.ServerVariables("HTTP_AUTHORIZATION"). After a little while I decided "why stop there?!" The tool will now check all the common pitfalls of Kerberos authentication except for a couple issues that I can't check for when the web request fails entirely (if my page won't even run, how can I check those settings?).

I'd like to hear your feedback!! If you have problems, constructive criticism, or praise please let me know. If you have comments or suggestions that will benefit the masses, I'll try to find time to implement your suggestions into the tool.

Usage Tips:

READ THE "Explanation" SECTIONS!! This tool is meant to be a teacher. If you don't read your material, you probably aren't going to understand what to do or at least why you're doing something that it suggests you do.
Did I mention that you should read the "Explanation" sections?? Do it!!
There are some "Fix This" buttons that help automate addressing issues. Although it isn't 100% clear in some of the output, it is probably easiest to use the "Fix This" buttons instead of manually following any KB articles that are also included in the output. You can, however, do whichever you prefer since either method should fix the applicable configuration problem.
If the DelegConfig tool doesn't work as expected, please see the "KNOWN PROBLEMS AND WORKAROUNDS" section below
If you end up making changes to your configuration based on what the tool reports, it is common that you'll need to log on/off the workstation machine and/or restart the IIS service or back-end service before you see Kerberos/Delegation start to work as desired.
Part of "Delegation" is authenticating with a back-end service. Don't forget to add the back-end server information with the applicable hyperlink to make sure everything looks good there too.

---------------------------------------------------------------
REQUIRED COMPONENTS:
---------------------------------------------------------------
1. All files in this ZIP are required for the ASPX page to operate properly.
2. ASP.NET version 1.1.4322.2300 or higher (also works with v2.0.50727) must be installed on the IIS server.

---------------------------------------------------------------
OPTIONAL COMPONENTS:
---------------------------------------------------------------
1. For more accurate results the .Net framework should be installed on the Workstation/Client

---------------------------------------------------------------
SETUP:
---------------------------------------------------------------
1. Unzip files to desired location that is *local* to the IIS server.
2. In the IIS MMC create a new virtual directory that points to the folder with the unzipped files.
3. Configure the virtual directory as an IIS application
a. In the IIS MMC right-click the vDir and choose Properties.
b. On the "Virtual Directory" tab click "Create" (if already configured as an IIS app you'll see a "Remove" button instead)
4. Ensure that "Scripts Only" (recommended) or "Scripts and Executables" is enabled for this vDir.

---------------------------------------------------------------
USAGE:
---------------------------------------------------------------
1. View the Default.aspx page through a web browser with an address such as "http://MyServer/MyVirtualDirectory/"

---------------------------------------------------------------
KNOWN PROBLEMS AND WORKAROUNDS.
---------------------------------------------------------------
General:
Generally speaking if there are any problems running this tool's Default.aspx page (errors of any sort) it is probably because Kerberos isn't working yet. :-p. In that case it is best to start by requesting the page locally from the IIS server. Certain types of problems only exist when IE attempts to connect to IIS using Kerberos and Kerberos is usually not used when local to the IIS server. Requesting the Default.aspx locally from the server will avoid many types of problems this DelegConfig tool can encounter. Once you can get this page working locally from the IIS server the tool will then report some of the problems that could affect Kerberos and/or Delegation. Once some of those items are addressed and Kerberos is closer to working you may be able to do additional checks by requesting the ASPX page from a remote IE machine.

--------------
Problem:
[HttpException (0x8007052e): Failed to start monitoring changes to '\\ServerName\ShareName'.]

ASP.net uses a hierarchical system of reading configuration files starting with Machine.Config, then web.config from the root of the web site, then finally web.config file from the virtual directory the ASPX application is running. Any time one of these files changes, IIS wants to be able to reload the file and restart the web application to ensure the most recent configuration is being used. If the dot net framework is not able to read from the root of the web site using the ASPNET account, this error will be returned. This is typical of when the home directory is a UNC path.

Workaround:
1. If running IIS 6.0 this can sometimes be fixed by disabling "Run www service in IIS 5.0 Isolation Mode".
OR
2. To ensure that ASPNET can read from the root of the web site, temporarily change the "Home Directory" of the site to a local path. Once you have resolved the Kerberos and Delegation issues based on the results of the ASPX application you can change the home directory back to the desired UNC path.

--------------
Problem:
Continuous password prompt with underlying 401.1 response. There are 2 common possibilities for this.

Additional Information:
Looking in the security event logs shows:

POSSIBILITY ONE (notice the Logon Process of Kerberos):
Event Type: Failure Audit
Event Source: Security
Event Category: Logon/Logoff
Event ID: 529
Date:  1/1/2005
Time:  6:00:00 PM
User:  NT AUTHORITY\SYSTEM
Computer: COMPUTERNAME
Description:
Logon Failure:
    Reason:  Unknown user name or bad password
    User Name:
    Domain:
    Logon Type: 3
   Logon Process: Kerberos
    Authentication Package: Kerberos

Workaround:
You receive an "HTTP Error 401.1 - Unauthorized: Access is denied due to invalid credentials" error message when you try to access a Web site that is part of an IIS 6.0 application pool
http://support.microsoft.com/?id=871179

POSSIBILITY TWO (notice the strange Logon Process):
Event Type: Failure Audit
Event Source: Security
Event Category: Logon/Logoff
Event ID: 537
Date:  1/1/2005
Time:  6:00:00 PM
User:  NT AUTHORITY\SYSTEM
Computer: COMPUTERNAME
Description:
Logon Failure:
    Reason:  An error occurred during logon
    User Name: someuser
    Domain:  SOMEDOMAIN
    Logon Type: 3
    Logon Process: Ðù²
    Authentication Package: NTLM
    Workstation Name: COMPUTERNAME
    Status code: 0xC000006D
    Substatus code: 0x0
    Source Network Address: 127.0.0.1

Workaround:
You receive error 401.1 when you browse a Web site that uses Integrated Authentication and is hosted on IIS 5.1 or IIS 6
http://support.microsoft.com/?id=896861