What's in your robots.txt?

[Nick]

Post what forum script you use (vBulletin, IP.Board, MyBB, phpBB, etc.) and what your robots.txt file consists of (to block search engines from crawling unnecessary pages).

[Michael]

vBulletin of course

User-agent: *
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Disallow: /login.php?do=lostpw
Disallow: /view.php?pg=advertise
Disallow: /home.php
Disallow: /arcade/
Disallow: /arcade.php
Disallow: /UTF/
Disallow: /sexes/
Disallow: /ads/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /register/

That is ours

[Kevinl]

Did I already do this here...or am I going crazy?

[Nick]

Originally Posted by Kevinl

Did I already do this here...or am I going crazy?

I think it was in another thread for something else.

[FullMetalBabe]

For MyBB

User-agent: msnbot
User-agent: googlebot
#Disallow: /showthread.php*$
Disallow: /member.php*$
Disallow: /search.php*$
Disallow: /memberlist.php*$
Disallow: /calendar.php*$
Disallow: /newreply.php*$
Disallow: /private.php*$
Disallow: /newthread.php
Disallow: /online.php
Disallow: /archive

User-agent: *
#Disallow: /showthread.php
Disallow: /member.php
Disallow: /search.php
Disallow: /memberlist.php
Disallow: /calendar.php
Disallow: /newreply.php
Disallow: /private.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /archive

[Dunhamzzz]

You guys should have a sitemap entry in there too.

[tryfuhl]

to disallow?

[Nick]

Originally Posted by tryfuhl

to disallow?

No, to point out where exactly your sitemap is located. Look up "sitemap AutoDiscovery".

Since working with Google and Microsoft to support a single format for submission with sitemaps, we have continued to discuss further enhancements to make it easy for webmasters to get their content to all search engines quickly.

All search crawlers recognize robots.txt, so it seemed like a good idea to use that mechanism to allow webmasters to share their sitemaps. You agreed and encouraged us to allow robots.txt discovery of sitemaps on our suggestion board. We took the idea to Google and Microsoft and are happy to announce today that you can now find your sitemaps in a uniform way across all participating engines. To do this, simply add the following line to your robots.txt file:

sitemap: http://www.example.tld/sitemap.xml

Add Sitemap To robots.txt For Autodiscovery

[Soliloquy]

Can you add more than one sitemap through the robots.txt? (And have it be recognized, that is.)

[Dan]

I love the WMW robots.txt blog - http://www.webmasterworld.com/robots.txt