A web site must not contain a robots.txt file.

Overview

Finding ID	Version	Rule ID	IA Controls	Severity
V-2260	WG310 IIS6	SV-28797r2_rule	ECLP-1	Medium

Description
Search engines are constantly at work on the Internet. Search engines are augmented by agents, often referred to as spiders or bots, which endeavor to capture and catalog web-site content. In turn, these search engines make the content they obtain and catalog available to any public web user. To request that a well behaved search engine not crawl and catalog a site, the web site may contain a file called robots.txt. This file contains directories and files that the web server SA desires not be crawled or cataloged, but this file can also be used, by an attacker or poorly coded search engine, as a directory and file index to a site. This information may be used to reduce an attacker’s time searching and traversing the web site to find files that might be relevant. If information on the web site needs to be protected from search engines and public view, other methods must be used.

Description

Search engines are constantly at work on the Internet. Search engines are augmented by agents, often referred to as spiders or bots, which endeavor to capture and catalog web-site content. In turn, these search engines make the content they obtain and catalog available to any public web user. To request that a well behaved search engine not crawl and catalog a site, the web site may contain a file called robots.txt. This file contains directories and files that the web server SA desires not be crawled or cataloged, but this file can also be used, by an attacker or poorly coded search engine, as a directory and file index to a site. This information may be used to reduce an attacker’s time searching and traversing the web site to find files that might be relevant. If information on the web site needs to be protected from search engines and public view, other methods must be used.

STIG	Date
IIS6 Site	2015-06-01

Details

Check Text ( C-30022r2_chk )
1. Open the IIS Manager > click on the web site being reviewed. 2. In the right hand pane look for a file named robots.txt. 3. If the robots.txt file does exist, this is a finding.

Fix Text (F-32685r2_fix)
1. Open the IIS Manager > click on the web site being reviewed. 2. In the right hand pane look for a file named robots.txt. 3. Delete the robots.txt file. NOTE: If there is information on the web site that needs protection from search engines and public view, then other methods must be used to safeguard the data.