The End of Metabase Corruption (R.I.P.)

Saturday, August 22, 2009

Even though IIS 7 with its new XML-based configuration settings has been around for a while, I was going through some old directories on one of my computers the other day and I stumbled across an image that I had labeled "Metabase Corruption." I have kept that image around for the past decade or so because it's one of the few actual examples of metabase corruption that I have actually verified, although there have definitely been more cases than just the ones that I have seen. But I thought that it might make for a good blog entry to explain the origins of the term "Metabase Corruption" and what that means to you if you're using a version of IIS that is earlier than IIS 7.

A brief history of the IIS metabase and metabase editors

Many years ago I was working in the Microsoft Product Support Services (PSS) group, where I provided support for enterprise-level customers that were using IIS versions 1.0 through 6.0. The early versions of IIS, (versions 1.0 through 3.0), stored all of their settings in the registry. This was fine for that period of time, but it didn't scale well and for several other reasons it was cumbersome to deal with. Just to bring back some nightmares, here's an image of the IIS 3.0 manager:

When IIS 4.0 was released, it introduced the "metabase," which was a proprietary database that stored all of the metadata for IIS, and all of that information was physically stored in a file that was named metabase.bin. The metabase scaled considerably better than the registry, but it introduced several other problems. The biggest problem was that the metabase was something of a "black box" for end users - all of your settings were stored in the metabase, and the only way to get to those settings was through the IIS Manager or by using unsupported command-line tools. The next big problem was that the metabase wasn't portable - you couldn't copy the metabase from one computer to another. (If you tried you would quickly realize that you just killed IIS.) When the IIS 4.0 Resource Kit was released it introduced MetaEdit 1.0, which was an editor for the metabase that looked like RegEdit, but it crashed so often that it really wasn't useful.

During the Windows 2000 timeframe I took over the MetaEdit code base and I created MetaEdit 2.0, which was released with the Windows 2000 Resource Kit. I fixed all of the stability problems from MetaEdit 1.0 and I added export/import functionality so that users could copy sections of the metabase between servers. Actually, I wrote MetaEdit before Microsoft stopped adding Easter Eggs to software, so there's a small Easter Egg in the Help::About for MetaEdit that you can find by holding down Shift-Ctrl-Alt and double-clicking the icon - you first see a photo of Bill Gates and then you see a photo of me from the days when I had longer hair...

I released a couple of patches to MetaEdit over the years, and the last version that I released is version 2.2, which you can download through the following URL:

How To Install MetaEdit 2.2 on Windows NT 4.0 or Windows 2000

When Windows Server 2003 was in development, XML had risen within the development community as a great way to store your data, so the IIS team decided to switch from the metabase.bin format to metabase.xml. This allowed for editing the metabase using any XML editor, even Windows Notepad, and IIS added native export and import functionality, so you could migrate settings from one computer to another by using IIS Manager.

It was sometime during the Windows 2003 timeframe that I was working on MetaEdit 3.0 when I heard that someone else was working on Metabase Explorer, and the early versions of Metabase Explorer that I was able to test were really cool, so I ceased work on MetaEdit in favor of Metabase Explorer, which was eventually released with the IIS 6.0 Resource Kit Tools. You can download the IIS 6.0 Resource Kit Tools through the following URL:

Internet Information Services (IIS) 6.0 Resource Kit Tools

Note: MetaEdit should only be used on Windows NT 4.0 with IIS 4.0 or Windows 2000 with IIS 5.0. For IIS 6.0 you should only use Metabase Explorer. (And yes - you can blame me for the totally useless dialog that you get when you try to use MetaEdit with IIS 6.0. Sorry about that. )

What's all this about metabase corruption?

It was shortly after the release of IIS 4.0 that I first heard the term "Metabase Corruption" being used. People were using the term "Metabase Corruption" as a catch-all phrase for "I don't what's broken but reinstalling IIS fixed the problem so it must have been a problem with the metabase." Everyone's general opinion seemed to be that if you couldn't find the problem and you can't look at the metabase, then it must be corruption in the metabase if you can't resolve the issue without reinstalling IIS. To me, this seemed like a cop-out for good troubleshooting, and since I was a technical lead in PSS around the time that IIS 5.0 was being released I would generally refuse to accept anyone's claim when they said that they had a customer with "Metabase Corruption." Instead, I would require them to troubleshoot the problem further, and in nearly 100% of those situations the problem was not caused by the metabase but by some other problem that could be rectified without reinstalling IIS.

Here's a case in point - several years ago I worked on an issue that some of the folks that supported SharePoint Portal Server 2001 on IIS 5.0 were seeing. They first described the problem to me as "IIS was losing settings." What they were actually seeing was that they would make changes in IIS Manager and apply them, but when they would close and reopen IIS Manager they saw that all their settings were gone. They claimed that this was "Metabase Corruption," and because of this they would often make the customer reinstall IIS completely. Needless to say I made them quite angry by stating that "Metabase Corruption" is almost never actual corruption, and that they were wasting the customer's time by making them reinstall IIS. Even though reinstalling IIS made the problem go away, it didn't actually fix the problem and it only hid the actual cause at the customer's expense.

Since I knew a little more than they did about how the metabase worked, I could theorize reasons why that behavior might occur and I asked them to contact me when they had a customer with this problem again. The next time they saw this problem I asked them to follow these steps:

Stop IIS by using "iisreset /stop" from a command-line
Start just the IISADMIN service by using "net start IISADMIN" from a command-line
Open the IIS Manager, make their changes, then close the IIS Manager
Restart IIS by using "iisreset /restart" from a command-line
Open IIS Manager and verify their settings

In 100% of the cases this enabled them to work around the issue, but that was only part of the solution. I theorized that something was causing IIS to crash before the metabase changes were flushed to disk, but the Windows service recovery settings were instructing the server to restart IIS whenever it crashed so it seemed to the customer like nothing had crashed at all. With this in mind, I had them check the Windows System Log in the Event Viewer, and it listed several errors indicating that the IISADMIN service had terminated unexpectedly, which confirmed my suspicions. After some additional troubleshooting I discovered that SharePoint Portal Server 2001 was setting several illegal values in the metabase, and sooner or later that would cause IIS 5.0 to crash. (SharePoint has long since fixed that problem.)

Does metabase corruption actually happen? If so, what can I do about it?

Sadly - yes it does. It doesn't happen as often as some people seem to think that it does, but it is certainly possible. The metabase is simply a file, and file corruption can occur for any number of reasons. In the early days of IIS 5.0 on the Windows 2000 beta I discovered (quite by accident) how I could overwrite one specific value in the metabase and crash IIS so badly that IIS couldn't even be uninstalled. (Don't worry - that problem has long since been fixed.) I have also seen metabase corruption due to hardware failures, like a drive controller that is going bad and the problem is manifesting itself slowly over time.

That being said, there are several methods to check if you actually have metabase corruption:

The first and easiest thing to check is - does IIS start and all of your web sites seem to work?
Can you open up the IIS Manager and see all of your sites?
Can you enumerate the metabase without errors using MetaEdit, Metabase Explorer, or "adsutil.vbs enum_all w3svc" from a command line?

If the answer to any or all of these questions is yes - then chances are good that you do not have metabase corruption, and whatever problem you have is due to misconfiguration or some other issue.

The best way that I can think of to prevent metabase corruption is to avoid using a variety of tools to constantly read and write to the metabase outside of normal IIS operations. I have written scripts that perform literally thousands of metabase operations under a nominal load in a lab environment for testing purposes and never had a problem, but a server under heavy load with thousands of read/write operations per minute could theoretically run into problems if left to run like that for an indefinite length of time. But theoretically the same may be true for any file under heavy read/write load for an indefinite length of time.

Protecting against metabase corruption

The best thing that you can do to prevent against metabase corruption is backup the metabase regularly!!! I cannot count the number of times where having a good backup of the metabase saved me from a myriad of issues. Fortunately, there are several KB articles that describe backing up the metabase:

The last KB article in the above list is based on a script that I wrote for one of my IIS 5.0 on Windows 2000 servers many years ago that I continue to use on any servers that run IIS 6.0 on Windows Server 2003.

Does IIS 7 have metabase corruption?

The quick answer to this is "no", but that answer is a little misleading since IIS 7 doesn't have a metabase!

In IIS 7 we rolled out a brand new XML-based configuration store, and the bulk of your IIS settings are stored in a file named applicationHost.config, which greatly resembles the web.config configuration paradigm that was established with the .NET framework. That being said, applicationHost.config is just another file, and even though its syntax is much easier to understand and manually edit, any file can theoretically become corrupted, so you should always make sure that you backup your applicationHost.config file as well.

With this in mind I wrote the following blog post some time ago where I discuss creating a script to automate backing up your IIS 7 configuration settings:

Automating IIS 7 Backups

(Note: I also wrote another blog post recently about how the script in that blog post saved my day when something did get corrupted on one of my servers.)

Parting thoughts

I started out this blog post by describing an image that I found on one of my computers where actual metabase corruption had occurred, so I thought that a nice way to close out this blog post would be to show you the actual image:

It's been almost ten years now, so I don't remember the root cause anymore, but I remember what it looked like. I had a mixture of sites that were working and others that were not, and when I opened the metabase in MetaEdit I noticed that scattered throughout the metabase were hundreds of values that shouldn't be there or values that did not make sense. MetaEdit was trying its best to show all of the data, but you'll notice in the highlighted areas of the image that several of the values had bad data in them.