Microsoft.com is the 5th most trafficked site in the world, 4th most in the US. The site is supported by roughly 80 Web servers delivering content daily to around 287M unique users via +300k concurrent connections at a rate of over 10k requests per second. Despite such a heavy load, Microsoft.com is consistently the #1 most available site on the Internet as measured by Keynote. The Operations team for MSCom had been playing with IIS7 since its early beta builds and recently finished deploying the new Web server in Windows Server 2008 Beta 3 to all its production servers.
To deploy a new operating system and new Web server into an environment like this without compromising performance or reliability is quite a feat. As the Technical Product Manager for IIS, I had to find out how they did it and more importantly, what they learned from this experience. To get this valuable info for IIS.NET, I sat down and had a chat with two of Microsoft.com’s top Operations Managers, Jeff Toewes and Chad Kraykovic. Lucky for you, the camera's were rolling.
You can view the 38 min video here (35MB) or for higher quality (296MB), here. So what did I learn from this interview? First and foremost, I learned how the Microsoft.com Ops team goes through the pain of deploying early builds of Microsoft’s new products so customers don’t have to. In addition to that, here are some of the most important take-aways from their IIS7 deployment that every Ops guy out there should be sure to know.What tips do the Microsoft.com guys have for customers considering the upgrade to IIS7?
So now that Microsoft.com has deployed IIS7, what benefits do they enjoy most? In this interview, they give some of the biggest highlights including:
What are the most important “gotchas” that slowed down MSCom during this deployment?
- Come up with a plan for deployment by talking to site stakeholders early to understand their specific needs and requirements
- Replay the requests in your logs using IIS7 in a pre-production lab to make sure that IIS7 works well for your specific scenarios. Tools like TinyGet and WCAT are extremely useful for this.
- Test your scripts. IIS7 has an IIS6 Metabase Compatibility layer to ensure your ADSI scripts just work. Still, test all your scripts anyway.
- Communicate any breaking changes to key site/content owners so they can assist with changes, particularly with respect to application configuration in IIS7 integrated mode.
What other IIS7 features is MSCom most excited to roll out next?· The delegated configuration capabilities in IIS7 will significantly reduce the administrative time MSCom invests in the site because they plan to set a delegation policy that will allow site and content owners to self-manage many of the most frequently requested tasks such as setting output caching in Web config, adding new redirects, or configuring custom errors. · Shared Web server configuration will allow MSCom to touch just a few shares to make changes to a bunch of boxes. · MSCom is seriously investigating dynamic compression in IIS7 because it is now powerful enough for the bandwidth savings and shortened response times to justify the cost of additional CPU overhead. · MSCom has already seen IIS7’s enhanced output caching bring performance for some dynamic sites up from 400 rps to 3000 rps. The team probably won't turn this on globally but in many instances, this feature will significantly improve application performance
- User Account Control (UAC) adds security but this new security model introduced in Windows Vista forced MSCom to adjust processes to perform administrative tasks, such as running of scripts in an elevated command prompt, etc.
- Module order in applicationHost.config matters because the order in which modules are listed is the order in which IIS loads them when processing requests. As an example, MSCom initially got bitten by failing to place the static file module last, behind modules for authentication and authorization.
- IIS7 Integrated mode can introduce several types of errors caused by specific breaking configuration changes in web.config files (in which IIS and ASP.NET settings now coexist), specifically around ASP.NET’s <handlers> and <modules> config sections. To maximize compatibility, use the Classic ASP.NET configuration option for your Application Pools, or migrate applications using the AppCmd.exe migrate option.
- To enhance IIS application security, Web server configuration is now scoped to the IIS 7 worker process that the application runs inside and as a result, MSCom could no longer query a central Metabase for physical path mappings, etc. This security win enables greater application isolation but impacted MSCOM due to some wide app dependencies on a few central physical folders.
- To enable compression, you now need to specify MIME-types instead of listing all file extensions to be compressed. Thus, some work was needed to map each of our file extensions to MIME type to get compression working.
- IIS Logs are in UNICODE format by default for IIS7, so log file names will start with “u_”. There are also slightly different logging behaviors, for example requests to slash used to include the default file, now it will log the explicit URL, so slash or default.aspx would be logged separately.
To learn more about how the Ops Team for MSCom uses IIS to host and manage one of the busiest sites in the world, check out their team blog at http://blogs.msdn.com/mscom.