Robots.txt

From mmmv_kos
Jump to: navigation, search


Iga alamdomeeni juurkataloogis peab olema eraldi robots.txt fail.


Süntaksikontroll

Aadressil http://tool.motoricerca.info/robots-checker.phtml olev tarkvara eeldab robots.txt täisrada.


Näited

User-agent: *

Disallow: /stat/
Disallow: /error/
Disallow: /cgi-bin/


User-agent: *
Disallow: /


User-agent: *
Disallow: /g/bonnet/
Disallow: /g/yellow_soap_opera_blog/user/


# For syntax checking, see:
# http://tool.motoricerca.info/robots-checker.phtml

User-agent: *
Crawl-delay: 10

# Directories
Disallow: /database/
Disallow: /includes/
Disallow: /misc/

# Files
Disallow: /xmlrpc.php
Disallow: /INSTALL.txt
Disallow: /INSTALL.mysql.txt