Search results matching tag 'IIS'

IIS URL Rewrite – rewriting non-www to www

Anonymous — Sat, 28 Nov 2009 01:55:32 GMT

If you’re using IIS 7.0 (or 7.5), URL Rewrite is a valuable tool, well worth installing and using.

One common use of URL Rewrite is redirecting http://domain.com to http://www.domain.com. Many people are doing this for search engine optimization (SEO) so that search engines only see the one site, rather than two sites. The goal is to set a permanent 301 redirect.

You can download URL Rewrite from http://www.iis.net/expand/URLRewrite. For this walkthrough and screenshots I’ll use URL Rewrite 2.0 RC1, but everything that I’ll cover also works for version 1.0 and 1.1.

URL Rewrite works at the global level, or site level (or application level for that matter). Where you apply it is really up to how you manage your server. Either will work for a domain name redirect like this.

You can choose to create the rules using IIS Manager, or using a text editor and updating web.config directly. I’ll show both, starting with IIS Manager.

Let’s get started. First, open IIS Manager and double-click on the “URL Rewrite” icon.

Next, click on “Add Rules…” from the Actions pane.

Here you’ll have a choice from a few wizard options, and with URL Rewrite 2.0 you can also create outbound rules. Create a Blank rule (inbound rules).

Give your rule a good friendly “Name”. I’ll call mine “Redirect domain.com to www”.

In the “Using” dropdown box you can choose between Regular Expressions and Wildcards. Use wildcards if you aren’t familiar with regular expressions since they are much more intuitive. However, if you later need to create more complex rules, regex may be necessary.

For this demo select Wildcards. However, I’ll include instructions for those wanting to use regular expressions.

Enter * for the “Pattern”. That means anything qualifies. We’ll use a condition later instead of matching to the URL. (for Regular Expressions, use .*).

Now expand the “Conditions” section and click “Add”. In the “Add Condition” dialogue enter the following:

Condition input:	{HTTP_HOST}
Check if input string:	Matches the Pattern
Pattern:	domain.com (for regex, enter ^domain.com$)
Ignore case:	checked

Click OK.

Finally, it’s time to set the Action.

In the Action section make sure that the “Action Type” is set to Redirect.

For the “Action Properties”, enter http://www.domain.com/{R:0}. The {R:0} retains the existing URL so if someone typed something like http://domain.com/aboutus it would retain the aboutus as it adds the www.

Be sure that the “Append query string” remains checked so that the querystring part is also retained.

Also, be sure that the “Redirect Type” is set to Permanent (301), which is what the search engines like. This tells the search engines to do a permanent redirect, use the new location and ignore the previous location.

Finally, Apply the rule and test!

Using a Text Editor

You can also create this rule manually by adding the following to your site’s web.config (or applicationHost.config if you set this at the server level).

In the <system.webServer> section of your web.config, add the following:

Wildcards

     <rewrite>

        <rules>

            <rule name="Redirect domain.com to www" patternSyntax="Wildcard" stopProcessing="true">

                <match url="*" />

                <conditions>

                    <add input="{HTTP_HOST}" pattern="domain.com" />

                </conditions>

                <action type="Redirect" url="http://www.domain.com/{R:0}" />

            </rule>

        </rules>

    </rewrite>

Save and you should be set.

Or, if you prefer Regular Expressions, use this instead:

Regular Expressions

    <rewrite>

        <rules>

            <rule name="Redirect domain.com to www" patternSyntax="ECMAScript" stopProcessing="true">

                <match url=".*" />

                <conditions>

                    <add input="{HTTP_HOST}" pattern="^domain.com$" />

                </conditions>

                <action type="Redirect" url="http://www.domain.com/{R:0}" />

            </rule>

        </rules>

    </rewrite>

This is just the start to great SEO, but it’s a common step and one that I hope you find helpful.

ODBC Logging, IIS 7.5, SQL Server 2008

Anonymous — Fri, 27 Nov 2009 03:32:00 GMT

I wrote an article in 2007 how to setup ODBC Logging on IIS 7.0. There was a post recently in the newsgroups someone wanted to setup and couldn't get it working. My article uses sql server 2005, IIS 7.0. I was able to reproduce an issue on my test server (Windows Server 2008 R2, x64). Here are some items I ran into.

Can't run the application pool can't run in 32 bit. This error came up

HTTP Error 500.0 - Internal Server Error

The page cannot be displayed because an internal server error has occurred.

Detailed Error Information

Module	CustomLoggingModule
Notification	LogRequest
Handler	StaticFile
Error Code	0x8007001f

Requested URL	http://localhost:8080/iisstart.htm
Physical Path	c:\inetpub\wwwroot\iisstart.htm
Logon Method	Anonymous
Logon User	Anonymous
Failed Request Tracing Log Directory	C:\inetpub

Can run the application pool using the ApplicationPoolIdentity (Default user)

Named Pipes needs to be enabled on sql server

Application Pool runs as a domain user, which has access to the database.

When you setup your DSN, make sure to explicitly test connection with the user setup in the ODBC logging config

Additional Steps to follow besides the original article

1) Follow my original setup

http://weblogs.asp.net/steveschofield/archive/2007/12/20/iis7-post-57-how-to-setup-odbc-logging-in-iis-7-0.aspx

2) Create a domain user

3) Setup the application pool to run as the domain user

4) Add user to the IIS_IUSRS group

5) Log into SQL Server, grant DBO or DB-Reader, DB-Writer permissions on InternetDB database

After doing these additional steps, I was able to run ODBC Logging and track information. As I mentioned in my previous article, ODBC logging is not meant to run on a busy website. "Millage may vary".

Hope this helps,

Steve Schofield
Microsoft MVP - IIS

IIS SEO Toolkit – Crawler Module Extensibility

Anonymous — Mon, 23 Nov 2009 22:55:00 GMT

Sample SEO Toolkit CrawlerModule Extensibility

In this blog we are going to write an example on how to extend the SEO Toolkit functionality, so for that we are going to pretend our company has a large Web site that includes several images, and now we are interested in making sure all of them comply to a certain standard, lets say all of them should be smaller than 1024x768 pixels and that the quality of the images is no less than 16 bits per pixel. Additionally we would also like to be able to make custom queries that can later allow us to further analyze the contents of the images and filter based on directories and more.

For this we will extend the SEO Toolkit crawling process to perform the additional processing for images, we will be adding the following new capabilities:

Capture additional information from the Content. In this case we will capture information about the image, in particular we will extend the report to add a "Image Width", "Image Height" and a "Image Pixel Format".
Flag additional violations. In this example we will flag three new violations:
1. Image is too large. This violation will be flagged any time the content length of the image is larger than the "Maximum Download Size per URL" configured at the start of the analysis. It will also flag this violation if the resolution is larger than 1024x768.
2. Image pixel format is too small. This violation will be flagged if the image is 8 or 4 bits per pixel.
3. Image has a small resolution. This will be flagged if the image resolution per inch is less than 72dpi.

Enter CrawlerModule

A crawler module is a class that extends the crawling process in Site Analysis to provide custom functionality while processing each URL. By deriving from this class you can easily raise your own set of violations or add your own data and links to any URL.

public abstract class CrawlerModule : IDisposable
{
    // Methods
    public virtual void BeginAnalysis();
    public virtual void EndAnalysis(bool cancelled);
    public abstract void Process(CrawlerProcessContext context);

    // Properties
    protected WebCrawler Crawler { get; }
    protected CrawlerSettings Settings { get; }
}

It includes three main methods:

BeginAnalysis. This method is invoked once at the beginning of the crawling process and allows you to perform any initialization needed. Common tasks include registering custom properties in the Report that can be accessed through the Crawler property.
Process. This method is invoked for each URL once its contents has been downloaded. The context argument includes a property URLInfo that provides all the metadata extracted for the URL. It also includes a list of Violations and Links in the URL. Common tasks include augmenting the metadata of the URL whether using its contents or external systems, flagging new custom Violations, or discovering new links in the contents.
EndAnalysis. This method is invoked once at the end of the crawling process and allows you to do any final calculations on the report once all the URLs have been processed. Common tasks in this method include performing aggregations of data across all the URLs, or identifying violations that depend on all the data being available (such as finding duplicates).

Coding the Image Crawler Module

Create a Class Library in Visual Studio and add the code shown below.

Open Visual Studio and select the option File->New Project
In the New Project dialog select the Class Library project template and specify a name and a location such as "SampleCrawlerModule"
Using the Menu "Project->Add Reference", add a reference to the IIS SEO Toolkit client library (C:\Program Files\Reference Assemblies\Microsoft\IIS\Microsoft.Web.Management.SEO.Client.dll).
Since we are going to be registering this through the IIS Manager extensibility, add a reference to the IIS Manager extensibility DLL (c:\windows\system32\inetsrv\Microsoft.Web.Management.dll) using the "Project->Add Reference" menu.
Also, since we will be using the .NET Bitmap class you need to add a reference to "System.Drawing" using the "Project->Add Reference" menu.
Delete the auto-generated Class1.cs since we will not be using it.
Using the Menu "Project->Add New Item" Add a new class named "ImageExtension".

using System;
using System.Drawing;
using System.Drawing.Imaging;
using Microsoft.Web.Management.SEO.Crawler;

namespace SampleCrawlerModule {

    /// <summary>
    /// Extension to add validation and metadata to images while crawling
    /// </summary>
    internal class ImageExtension : CrawlerModule {
        private const string ImageWidthField = "iWidth";
        private const string ImageHeightField = "iHeight";
        private const string ImagePixelFormatField = "iPixFmt";

        public override void BeginAnalysis() {
            // Register the properties we want to augment at the begining of the analysis
            Crawler.Report.RegisterProperty(ImageWidthField, "Image Width", typeof(int));
            Crawler.Report.RegisterProperty(ImageHeightField, "Image Height", typeof(int));
            Crawler.Report.RegisterProperty(ImagePixelFormatField, "Image Pixel Format", typeof(string));
        }

        public override void Process(CrawlerProcessContext context) {
            // Make sure only process the Content Types we need to
            switch (context.UrlInfo.ContentTypeNormalized) {
                case "image/jpeg":
                case "image/png":
                case "image/gif":
                case "image/bmp":
                    // Process only known content types
                    break;
                default:
                    // Ignore any other
                    return;
            }

            //--------------------------------------------
            // If the content length of the image was larger than the max
            //   allowed to download, then flag a violation, and stop
            if (context.UrlInfo.ContentLength >
                Crawler.Settings.MaxContentLength) {
                Violations.AddImageTooLargeViolation(context,
                    "It is larger than the allowed download size");
                // Stop processing since we do not have all the content
                return;
            }

            // Load the image from the response into a bitmap
            using (Bitmap bitmap = new Bitmap(context.UrlInfo.ResponseStream)) {
                Size size = bitmap.Size;

                //--------------------------------------------
                // Augment the metadata by adding our fields
                context.UrlInfo.SetPropertyValue(ImageWidthField, size.Width);
                context.UrlInfo.SetPropertyValue(ImageHeightField, size.Height);
                context.UrlInfo.SetPropertyValue(ImagePixelFormatField, bitmap.PixelFormat.ToString());

                //--------------------------------------------
                // Additional Violations:
                //
                // If the size is outside our standards, then flag violation
                if (size.Width > 1024 &&
                    size.Height > 768) {
                    Violations.AddImageTooLargeViolation(context,
                        String.Format("The image size is: {0}x{1}",
                                      size.Width, size.Height));
                }

                // If the format is outside our standards, then flag violation
                switch (bitmap.PixelFormat) {
                    case PixelFormat.Format1bppIndexed:
                    case PixelFormat.Format4bppIndexed:
                    case PixelFormat.Format8bppIndexed:
                        Violations.AddImagePixelFormatSmall(context);
                        break;
                }

                if (bitmap.VerticalResolution <= 72 ||
                    bitmap.HorizontalResolution <= 72) {
                    Violations.AddImageResolutionSmall(context,
                        bitmap.HorizontalResolution + "x" + bitmap.VerticalResolution);
                }
            }
        }

        /// <summary>
        /// Helper class to hold the violations
        /// </summary>
        private static class Violations {

            private static readonly ViolationInfo ImageTooLarge =
                new ViolationInfo("ImageTooLarge",
                                  ViolationLevel.Warning,
                                  "Image is too large.",
                                  "The Image is too large: {details}.",
                                  "Make sure that the image content is required.",
                                  "Images");

            private static readonly ViolationInfo ImagePixelFormatSmall =
                new ViolationInfo("ImagePixelFormatSmall",
                                  ViolationLevel.Warning,
                                  "Image pixel format is too small.",
                                  "The Image pixel format is too small",
                                  "Make sure that the quality of the image is good.",
                                  "Images");

            private static readonly ViolationInfo ImageResolutionSmall =
                new ViolationInfo("ImageResolutionSmall",
                                  ViolationLevel.Warning,
                                  "Image resolution is small.",
                                  "The Image resolution is too small: ({res})",
                                  "Make sure that the image quality is good.",
                                  "Images");

            internal static void AddImageTooLargeViolation(CrawlerProcessContext context, string details) {
                context.Violations.Add(new Violation(ImageTooLarge,
                        0, "details", details));
            }

            internal static void AddImagePixelFormatSmall(CrawlerProcessContext context) {
                context.Violations.Add(new Violation(ImagePixelFormatSmall, 0));
            }

            internal static void AddImageResolutionSmall(CrawlerProcessContext context, string resolution) {
                context.Violations.Add(new Violation(ImageResolutionSmall,
                        0, "res", resolution));
            }
        }
    }
}

As you can see in the BeginAnalysis the module registers three new properties with the Report using the Crawler property. This is only required if you want to provide either a custom text or use it for different type other than a string. Note that current version only allows primitive types like Integer, Float, DateTime, etc.

During the Process method it first makes sure that it only runs for known content types, then it performs any validations raising a set of custom violations that are defined in the Violations static helper class. Note that we load the content from the Response Stream, which is the property that contains the received from the server. Note that if you were analyzing text the property Response would contain the content (this is based on Content Type, so HTML, XML, CSS, etc, will be kept in this String property).

Registering it

When running inside IIS Manager, crawler modules need to be registered as a standard UI module first and then inside their initialization they need to be registered using the IExtensibilityManager interface. In this case to keep the code as simple as possible everything is added in a single file. So add a new file called "RegistrationCode.cs" and include the contents below:

using System;
using Microsoft.Web.Management.Client;
using Microsoft.Web.Management.SEO.Crawler;
using Microsoft.Web.Management.Server;

namespace SampleCrawlerModule {
    internal class SampleCrawlerModuleProvider : ModuleProvider {
        public override ModuleDefinition GetModuleDefinition(IManagementContext context) {
            return new ModuleDefinition(Name, typeof(SampleCrawlerModule).AssemblyQualifiedName);
        }

        public override Type ServiceType {
            get { return null; }
        }

        public override bool SupportsScope(ManagementScope scope) {
            return true;
        }
    }

    internal class SampleCrawlerModule : Module {
        protected override void Initialize(IServiceProvider serviceProvider, ModuleInfo moduleInfo) {
            base.Initialize(serviceProvider, moduleInfo);

            IExtensibilityManager em = (IExtensibilityManager)GetService(typeof(IExtensibilityManager));
            em.RegisterExtension(typeof(CrawlerModule), new ImageExtension());
        }
    }
}

This code defines a standard UI IIS Manager module and in its client-side initialize method it uses the IExtensibilityManager interface to register the new instance of the Image extension. This will make it visible to the Site Analysis feature.

Testing it

To test it we need to add the UI module to Administration.config, that also means that the assembly needs to be registered in the GAC.

To Strongly name the assembly

In Visual Studio, you can do this easily by using the menu "Project->Properties", and select the "Signing" tab, check the "Sign the assembly", and choose a file, if you don't have one you can easily just choose New and specify a name.

After this you can compile and now should be able to add it to the GAC.

To GAC it

If you have the SDK's you should be able to call it like in my case:

"\Program Files\Microsoft SDKs\Windows\v6.0A\bin\gacutil.exe" /if SampleCrawlerModule.dll

(Note, you could also just open Windows Explorer, navigate to c:\Windows\assembly and drag & drop your file in there, that will GAC it automatically).

Finally to see the right name that should be use in Administration.config run the following command:

"\Program Files\Microsoft SDKs\Windows\v6.0A\bin\gacutil.exe" /l SampleCrawlerModule

In my case it displays:

SampleCrawlerModule, Version=1.0.0.0, Culture=neutral, PublicKeyToken=6f4d9863e5b22f10, …

Finally register it in Administration.config

Open Administration.config in Notepad using an elevated instance, find the </moduleProviders> and add a string like the one below but replacing the right values for Version and PublicKeyToken:

Use it

After registration you now should be able to launch IIS Manager and navigate to Search Engine Optimization. Start a new Analysis to your Web site. Once completed if there are any violations you will see them correctly in the Violations Summary or any other report. For example see below all the violations in the "Images" category.

Since we also extended the metadata by including the new fields (Image Width, Image Height, and Image Pixel Format) now you can use them with the Query infrastructure to easily create a report of all the images:

And since they are standard fields, they can be used in Filters, Groups, and any other functionality, including exporting data. So for example the following query can be opened in the Site Analysis feature and will display an average of the width and height of images summarized by type of image:

<?xml version="1.0" encoding="utf-8"?>
<query dataSource="urls">
<filter>
    <expression field="ContentTypeNormalized" operator="Begins" value="image/" />
</filter>
<group>
    <field name="ContentTypeNormalized" />
</group>
<displayFields>
    <field name="ContentTypeNormalized" />
    <field name="(Count)" />
    <field name="Average(iWidth)" />
    <field name="Average(iHeight)" />
</displayFields>
</query>

And of course violation details are shown as specified, including Recommendation, Description, etc:

Summary

As you can see extending the SEO Toolkit using a Crawler Module allows you to provide additional information, whether Metadata, Violations or Links to any document being processed. This can be used to add support for content types not supported out-of-the box such as PDF, Office Documents or anything else that you need. It also can be used to extend the metadata by writing custom code to wire data from other system into the report giving you the ability to exploit this data using the Query capabilities of Site Analysis.

IIS SEO Toolkit Extensibility

Anonymous — Mon, 23 Nov 2009 21:46:56 GMT

The IIS SEO Toolkit includes a lot of functionality built-in such as built-in violation rules, processing of different content types (like HTML, CSS, RSS, etc) and more, however it might not do all the things that you would need it to do, for example, it might not process a set of documents that you use, or it might not gather all the information that you are interested in while processing a document. The good news is that it includes enough extensibility to let you build on top of its rich capabilities and provide additional ones easily using .NET.

There are three main extensibility points in this first release, including:

Crawler Module. This extensibility point allows you to provide your own code to hook to the process of crawling a Web site in the Site Analysis process. Using this extensibility point you can extend the built in set of violation rules with your own. You can also gather additional information such as links or any metadata of a resource, whether directly extracted from the content or from an external system.
Site Analyzer Extension. This feature allows you to provide your own set of tasks to be exposed in the Site Analysis user interface. These tasks will be displayed in the main menu bar in the report dashboard.
Sitemap Extension. This class allows you to provide your own set of tasks to be exposed in the Sitemaps, Sitemap and Sitemap Index user interface.

This is the first of a series of extensibility blog entries for the IIS SEO Toolkit where I will cover all of the extensibility points mentioned above.

Slides for IIS – Web Application Gallery presentation at DevConnections

Anonymous — Mon, 23 Nov 2009 20:55:44 GMT

Two weeks ago I presented at DevConnections the talk "AMS10: Developing and Deploying for the Windows Web App Gallery", here are the slides.

Download the Web Application Gallery Talk slides here.

A few final links:

Microsoft Web Platform: http://www.microsoft.com/web/

Download Web PI: http://www.microsoft.com/web/downloads/platform.aspx

Submit your Applications at: http://www.microsoft.com/web/gallery/developer.aspx

IIS SEO Toolkit - New Reports (Redirects and Link Depth)

Anonymous — Wed, 18 Nov 2009 23:33:00 GMT

In the new version of the IIS SEO Toolkit we added two new reports that are very interesting, both from an SEO perspective as well as from user experience and site organization. These reports are located in the Links category of the reports

Redirects

This report shows a summary of all the redirects that were found while crawling the Web site. The first column (Linking-URL) is the URL that was visited that resulted in redirection to the Linked-URL (second column). The third column (Linking-Status code) specifies what type of redirection happened based on the HTTP status code enumeration. The most common values will be MovedPermanently/Moved which is a 301, or Found/Redirect which is a 302. The last column shows the status code for the final URL so you can easily identify redirects that failed or that redirected to another redirect.

Why should you care

This report is interesting because Redirects might affect your Search Engine rankings and make your users have the perception that your site is slower. For more information on Redirects see: Redirects, 301, 302 and IIS SEO Toolkit

Link Depth

This is probably one of my favorite reports since it is almost impossible to find this type of information in any other 'easy' way.

The report basically tells you how hard it is for users that land in your home page to get to any of the pages in your site. For example in the image below it shows that it takes 5 clicks for a user to get from the home page of my site to the XGrid.htc component.

This is very valuable information because you will be able to understand how deep your Web site is, in my case if you were to walk the entire site and layout its structure in a hierarchical diagram it would basically be 5 levels deep. Remember, you want your site to be shallow so that its easily discoverable and crawled by Search Engines.

Even more interesting you can double click any of the results and see the list of clicks that the user has to make it to get to the page.

Note that it shows the URL, the Title of the page as well as the Text of the Link you need to click to get to the Next URL (the one with a smaller index). So as you can see in my case the user needs to go to the home page, click the link with text "XGrid", which takes it to the /XGrid/ url (index 3) which then needs to click the link with text "This is a new...", etc.

Note that as you select the URLs in the list it will highlight in the markup the link that takes you to the next URL.

The data of this report is powered by a new type of query we called Route Query. The reason this is interesting is because you can customize the report to add different filters, or change the start URL, or more.

For example, lets say I want to figure out all the pages that the user can get to when they land in my site in a specific page, say http://www.carlosag.net/Tools/XGrid/editsample.htm:

In the Dashboard view of a Report, select the option 'Query->New Routes Query'. This will open a new Query tab where you can specify the Start URL that you are interested.

As you can see this report clearly shows that if a user visits my site and lands on this page they will basically be blocked and only be able to see 8 pages of the entire site. This is a clear example on where a link to the Home page would be beneficial.

Other common scenarios that this query infrastructure could be used for is to find ways to direct traffic from your most common pages to your conversion pages, this report will let you figure out how difficult or easy it is to get from any page to your conversion pages

IIS SEO Toolkit - Start new analysis automatically through code

Anonymous — Wed, 18 Nov 2009 06:00:28 GMT

One question that I've been asked several times is: "Is it possible to schedule the IIS SEO Toolkit to run automatically every night?". Other related questions are: "Can I automate the SEO Toolkit so that as part of my build process I'm able to catch regressions on my application?", or "Can I run it automatically after every check-in to my source control system to ensure no links are broken?", etc.

The good news is that the answer is YES!. The bad news is that you have to write a bit of code to be able to make it work. Basically the SEO Toolkit includes a Managed code API to be able to start the analysis just like the User Interface does, and you can call it from any application you want using Managed Code.

In this blog I will show you how to write a simple command application that will start a new analysis against the site provided in the command line argument and process a few queries after finishing.

IIS SEO Crawling APIs

The most important type included is a class called WebCrawler. This class takes care of all the process of driving the analysis. The following image shows this class and some of the related classes that you will need to use for this.

The WebCrawler class is initialized through the configuration specified in the CrawlerSettings. The WebCrawler class also contains two methods Start() and Stop() which starts the crawling process in a set of background threads. With the WebCrawler class you can also gain access to the CrawlerReport through the Report property. The CrawlerReport class represents the results (whether completed or in progress) of the crawling process. It has a method called GetUrls() that returns an instance to all the UrlInfo items. A UrlInfo is the most important class that represents a URL that has been downloaded and processed, it has all the metadata such as Title, Description, ContentLength, ContentType, and the set of Violations and Links that it includes.

Developing the Sample

Start Visual Studio.
Select the option "File->New Project"
In the "New Project" dialog select the template "Console Application", enter the name "SEORunner" and press OK.
Using the menu "Project->Add Reference" add a reference to the IIS SEO Toolkit Client assembly "c:\Program Files\Reference Assemblies\Microsoft\IIS\Microsoft.Web.Management.SEO.Client.dll".
Replace the code in the file Program.cs with the code shown below.
Build the Solution

using System;
using System.IO;
using System.Linq;
using System.Net;
using System.Threading;
using Microsoft.Web.Management.SEO.Crawler;

namespace SEORunner {
    class Program {

        static void Main(string[] args) {

            if (args.Length != 1) {
                Console.WriteLine("Please specify the URL.");
                return;
            }

            // Create a URI class
            Uri startUrl = new Uri(args[0]);

            // Run the analysis
            CrawlerReport report = RunAnalysis(startUrl);

            // Run a few queries...
            LogSummary(report);

            LogStatusCodeSummary(report);

            LogBrokenLinks(report);
        }

        private static CrawlerReport RunAnalysis(Uri startUrl) {
            CrawlerSettings settings = new CrawlerSettings(startUrl);
            settings.ExternalLinkCriteria = ExternalLinkCriteria.SameFolderAndDeeper;
            // Generate a unique name
            settings.Name = startUrl.Host + " " + DateTime.Now.ToString("yy-MM-dd hh-mm-ss");

            // Use the same directory as the default used by the UI
            string path = Path.Combine(
                Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments),
                "IIS SEO Reports");

            settings.DirectoryCache = Path.Combine(path, settings.Name);

            // Create a new crawler and start running
            WebCrawler crawler = new WebCrawler(settings);
            crawler.Start();

            Console.WriteLine("Processed - Remaining - Download Size");
            while (crawler.IsRunning) {
                Thread.Sleep(1000);
                Console.WriteLine("{0,9:N0} - {1,9:N0} - {2,9:N2} MB",
                    crawler.Report.GetUrlCount(),
                    crawler.RemainingUrls,
                    crawler.BytesDownloaded / 1048576.0f);
            }

            // Save the report
            crawler.Report.Save(path);

            Console.WriteLine("Crawling complete!!!");

            return crawler.Report;
        }

        private static void LogSummary(CrawlerReport report) {
            Console.WriteLine();
            Console.WriteLine("----------------------------");
            Console.WriteLine(" Overview");
            Console.WriteLine("----------------------------");
            Console.WriteLine("Start URL: {0}", report.Settings.StartUrl);
            Console.WriteLine("Start Time: {0}", report.Settings.StartTime);
            Console.WriteLine("End Time:   {0}", report.Settings.EndTime);
            Console.WriteLine("URLs:       {0}", report.GetUrlCount());
            Console.WriteLine("Links:      {0}", report.Settings.LinkCount);
            Console.WriteLine("Violations: {0}", report.Settings.ViolationCount);
        }

        private static void LogBrokenLinks(CrawlerReport report) {
            Console.WriteLine();
            Console.WriteLine("----------------------------");
            Console.WriteLine(" Broken links");
            Console.WriteLine("----------------------------");
            foreach (var item in from url in report.GetUrls()
                                 where url.StatusCode == HttpStatusCode.NotFound &&
                                       !url.IsExternal
                                 orderby url.Url.AbsoluteUri ascending
                                 select url) {
                Console.WriteLine(item.Url.AbsoluteUri);
            }
        }

        private static void LogStatusCodeSummary(CrawlerReport report) {
            Console.WriteLine();
            Console.WriteLine("----------------------------");
            Console.WriteLine(" Status Code summary");
            Console.WriteLine("----------------------------");
            foreach (var item in from url in report.GetUrls()
                                 group url by url.StatusCode into g
                                 orderby g.Key
                                 select g) {
                Console.WriteLine("{0,20} - {1,5:N0}", item.Key, item.Count());
            }
        }
    }
}

If you are not using Visual Studio, you can just save the contents above in a file, call it SEORunner.cs and compile it using the command line:

C:\Windows\Microsoft.NET\Framework\v3.5\csc.exe /r:"c:\Program Files\Reference Assemblies\Microsoft\IIS\Microsoft.Web.Management.SEO.Client.dll" /optimize+ SEORunner.cs

After that you should be able to run SEORunner.exe and pass the URL of your site as a argument, you will see an output like:

Processed - Remaining - Download Size
       56 -       149 -      0.93 MB
      127 -       160 -      2.26 MB
      185 -       108 -      3.24 MB
      228 -        72 -      4.16 MB
      254 -        48 -      4.98 MB
      277 -        36 -      5.36 MB
      295 -        52 -      6.57 MB
      323 -        25 -      7.53 MB
      340 -         9 -      8.05 MB
      358 -         1 -      8.62 MB
      362 -         0 -      8.81 MB
Crawling complete!!!

----------------------------
 Overview
----------------------------
Start URL:  http://www.carlosag.net/
Start Time: 11/16/2009 12:16:04 AM
End Time:   11/16/2009 12:16:15 AM
URLs:       362
Links:      3463
Violations: 838

----------------------------
 Status Code summary
----------------------------
                  OK -   319
    MovedPermanently -    17
               Found -    23
            NotFound -     2
 InternalServerError -     1

----------------------------
 Broken links
----------------------------
http://www.carlosag.net/downloads/ExcelSamples.zip

The most interesting method above is RunAnalysis, it creates a new instance of the CrawlerSettings and specifies the start URL. Note that it also specifies that we should consider internal all the pages that are hosted in the same directory or subdirectories. We also set the a unique name for the report and use the same directory as the IIS SEO UI uses so that opening IIS Manager will show the reports just as if they were generated by it. Then we finally call Start() which will start the number of worker threads specified in the WebCrawler::WorkerCount property. We finally just wait for the WebCrawler to be done by querying the IsRunning property.

The remaining methods just leverage LINQ to perform a few queries to output things like a report aggregating all the URLs processed by Status code and more.

Summary

As you can see the IIS SEO Toolkit crawling APIs allow you to easily write your own application to start the analysis against your Web site which can be easily integrated with the Windows Task Scheduler or your own scripts or build system to easily allow for continuous integration.

Once the report is saved locally it can then be opened using IIS Manager and continue further analysis as with any other report. This sample console application can be scheduled using the Windows Task Scheduler so that it can run every night or at any time. Note that you could also write a few lines of PowerShell to automate it without the need of writing C# code and do that by only command line, but that is left for another post.

Announcing: IIS SEO Toolkit v1.0 release

Anonymous — Tue, 17 Nov 2009 21:06:21 GMT

Today we are announcing the final release of the IIS Search Engine Optimization (SEO) Toolkit v1.0. This version builds upon the Beta 1 and Beta 2 versions and is 100% compatible with those versions so any report you currently have continues to work in the new version. The new version includes a set of bug fixes and new features such as:

Extensibility. In this version we are opening a new set of API's to allow you to develop extensions for the crawling process, including the ability to augment the metadata in the report with your own, extend the set of tasks provided in the Site Analysis and Sitemaps User Interface and more. More on this on a upcoming post.
New Reports. Based on feedback we added a Redirects summary report in the Links section as well as a new Link Depth report that allows you to easily know which pages are the "most hidden pages" in your site, or in other words if a user landed at your sites home page, "how many clicks does he need to do to reach a particular page".
New Routes Query. We added a new type of Query called Routes. This is the underlying data that powers the "Link Depth" report mentioned above, however it is also exposed as a new query type so that you can create your own queries to customize the Start page and any other kind of things, like filtering, grouping, etc.
New option to opt-out from keeping a local cache of files. We added a new switch in the "Advanced Settings" of the New Analysis dialog to disable the option of keeping the files stored locally. This allows you to run a report which runs faster and that consumes a lot less disk space than when keeping the files cached. The only side effect is that you will not be able to get the "Content" tab and the contextual position of the links as well as the Word Analysis feature. Everything else continues to work just as any other report.
HTML Metadata is now stored in the Report. By leveraging the Extensibility mentioned in bullet 1, the HTML parser now stores all the HTML META tags content so that you can later use them to write your own queries, whether to filter, group data or just export it, this gives you a very interesting set of options if you have any metadata like Author, or any custom.
Several Bug Fixes:

Internal URLs linked by External URLs now are also included in the crawling process.
Groupings in queries should be case sensitive
Show contextual information (link position) in Routes
The Duplicate detection logic should only include valid responses (do not include 404 NOT Found, 401, etc)
Canonical URLs should support sub-domains.
Several Accessibility fixes. (High DPI, Truncation in small resolutions, Hotkeys, Keyboard navigation, etc).
Several fixes for Right-To-Left languages. (Layout and UI)
Help shortcuts enabled.
New Context Menus for Copying content
Add link position information for Canonical URLs
Remove x-javascript validation for this release
Robots algorithm should be case sensitive
many more

This version can upgrade both Beta 1 and Beta 2 version so go ahead and try it and PLEASE provide us with feedback and any additional things you would like to see for the next version at the SEO Forum in the IIS Web site.

Click here to install the IIS SEO Toolkit.

Improve Web User Experience with IIS SEO Toolkit RTW and IIS URL Rewriter 2.0 RC

Anonymous — Tue, 17 Nov 2009 17:00:00 GMT

Remember the scene in Diehard when he shoots the glass? That was cool. What’s even cooler is now you can fine tune your Web site for the search engine, thus improving the experience for your Web users.

Today, Microsoft announced the releases of IIS Search Engine Optimization (SEO) Toolkit RTW and IIS URL Rewriter 2.0 RC – both extensions for IIS.

Don’t take it personally – I know it can be hard when your significant other tells you how to make improvements on something you worked hard on, let alone a faceless piece of software. But really, they are both just trying to help. Think of the customers.

The IIS SEO Toolkit helps Web developers, hosting providers, and Web server administrators to improve their Web site’s relevance in search results by recommending how to make the site content more search engine-friendly. The IIS SEO Toolkit helps to improve the volume and quality of traffic to your Web site from search engines, control how search engines access and display Web content, and inform search engines about locations that are available for indexing.

We’ve all been to a cocktail party when you’re trying to impress that certain person, and they ask for the URL of the latest project you are working on. It’s embarrassing when you have to rattle off a long-winded URL with question marks, equal signs, random ID numbers, and ampersands. But with URL Rewriter, you don’t have to anymore.

IIS URL Rewriter 2.0 adds support for outbound response rewriting and enables Web administrators to create powerful rules to implement URLs that are easier for users to remember and easier for search engines to find. IIS URL Rewriter 2.0 can now replace the URLs generated by a Web application in the response HTML with a more user friendly and search engine friendly equivalent, modify the links in the HTML markup generated by a Web application behind a reverse proxy, and fix up the content of any HTTP response by using regular expression pattern matching. IIS URL Rewriter 2.0 helps to easily define rules that match URLs or HTTP headers to generate more friendly and consistent URLs, protect content and assets from unauthorized linking and scanning, and integrate with existing IIS features to improve management, performance, and troubleshooting.

Did I mention it’s free? And it’s available to download and install today with the Microsoft Web Platform Installer 2.0, your one-stop shop to get everything for the Microsoft Web Platform – including IIS SEO Toolkit RTW and IIS URL Rewriter 2.0 RC. From IIS Extensions to the Windows Web App Gallery and beyond, there is sure to be something useful for your Web site endeavors.

Oh, if you run PHP, be sure to check out our learning section on Running PHP Applications on IIS. We have a ton of brand new and updated content to help provide PHP on Windows users a single source of end-to-end guidance. Pretty sweet!

For more information or to download, please visit the IIS SEO Toolkit RTW page and the IIS URL Rewriter 2.0 RC page.

Eric Rezabek

Senior Product Manager

IIS/Web

Streaming Media West

Anonymous — Mon, 16 Nov 2009 18:38:41 GMT

This week, I'm at Streaming Media West in San Jose, California. Seeing the sun, blue skies and relatively warm weather is a pleasant change from the rain swept northwest.
Today is a special pre-conference half-day workshop on Live Broadcasting with Silverlight and Windows Media, which my colleague Chris Knowlton is running. We're videotaping it for posterity and will get it posted as soon as we can.
On Wednesday morning, Marc Whitten, GM of Xbox Live will be delivering the keynote at 9am and on Wednesday afternoon, Steven Woodward from Microsoft and Mike Moser from Vertigo will be delivering a special session on real-world Smooth Streaming using a very recent and timely event as an example.
The exhibition runs Tuesday evening through Thursday afternoon and we'll have a special 20 minute presentation of Smooth Streaming on the show floor on Thursday at 1pm.
If you want to attend the conference or exhibition and don't yet have a pass, you can register at a discounted rate by going to https://secure.infotoday.com/forms/default.aspx?form=smw2009 and using the VIPMSFT as the promo code.