Using the IIS 7 URL Rewrite Module to block crawlers

Here’s an easy way to block the main web crawlers – Google Bing and Yahoo – from indexing any site across an entire server. This is really useful if you push all your beta builds to a public facing server, but don’t want them indexed yet by the search engines.

1. Install the IIS URL Rewrite Module.

2. At the server level, add a request blocking rule. Block user-agent headers matching the regex: googlebot|msnbot|slurp.

Or, just paste this rule into “C:\Windows\System32\inetsrv\config\applicationHost.config”

<system.webServer>
   <rewrite>
      <globalRules>
         <rule name="RequestBlockingRule1" stopProcessing="true">
            <match url=".*" />
            <conditions>
               <add input="{HTTP_USER_AGENT}" pattern="googlebot|msnbot|slurp" />
            </conditions>
            <action type="CustomResponse" statusCode="403"
               statusReason="Forbidden: Access is denied."
               statusDescription="You do not have permission to view this page." />
         </rule>
      </globalRules>
   </rewrite>
</system.webServer>

This’ll block Google, Bing and Yahoo from indexing any site published on the server. To test it out, try the Firefox User Agent Switcher.