Robots.txt

[homer]

Search engines behave differently upon encountering, or not, the robots.txt file during a crawl. You only have to follow your web stats to know that the robots.txt is one of the most requested files by search engine spiders. Many spiders check the robots.txt file first before ever performing a crawl, and some even pre-empt crawls by checking for the presence of, and commands in, the file; only to leave and come back another day.

[mikelbeck]

Good information there. Since every site is different, the robots.txt file will be signifigantly different based on how the site is set up.

However, for forum sites, such as phpBB or vBulletin, there are "default" robot.txt files that will generally disallow access to certain directories that a robot would have no reason for being in, such as the admin panel, includes files, etc. Most CMS systems come with a similar set up.

[crash]

Yeah I've read in a forum about someone wanting to delete robot.txt because it uses to much bandwith. Is that a good idea?

[mikelbeck]

Originally Posted by crash

Yeah I've read in a forum about someone wanting to delete robot.txt because it uses to much bandwith. Is that a good idea?

That doesn't even make any sense. The robots.txt file doesn't use any bandwidth.

[crash]

Well i read. It didn't happen to me. I just thought it was of interest!

[dotcomguy2]

yeh i also read it ... what crash read i think was that a guy was getting high bandwith from the SE's so he wanted too decrease it so he asked if i should delete the robot.txt but as you would expect people told him not too

hope this clears it up

[crash]

Yeah. Thanks dotcomguy2. I still don't know if it was real though. Does it really use bandwith?

[kiz9999]

No it doesn't use bandwith and under no circumstances should you ever delete it!

[forumowner]

yeh too right kiz9999. you should never ever delete robot.txt ever! period.

[Jenita]

Ya its true some search engines first check the robots.txt while crawling the sites.