Admin
Joined: 08 Apr 2007
Posts: 51
|
Learn how to use robots.txt to control how search engines and crawlers access your site.
# disallow all crawlers
User-agent: *
# disallow a specific crawler
User-agent: GoogleBot
# disallow all files
Disallow: /
# disallow specific urls
Disallow: /login.php
Disallow: /support/faq.html
Disallow: /contact.asp
# disallow all files with a ? in url
Disallow: /*?*
# disallow all files ending with these extensions
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.txt$
# disallow all files in a specific directory
Disallow: /admin/
Disallow: /comments/
# disallow archiving site
User-agent: ia_archiver
Disallow: /
# allow google image bot to search all images
User-agent: Googlebot-Image
Allow: /*
Note: The robots.txt file must be uploaded to the root accessible directory of your site (ie: mysite.com/robots.txt)
_________________
Please do not email or PM me with support questions. Please post them on the forum instead.
 |
|
