# robots.txt file to exclude files from access by web robots User-agent: * Disallow: /_private/ Disallow: /admin/ Disallow: /anonymous/ #Allow: /BrowserNews/ Disallow: /cgi-bin/ Disallow: /images/ #Allow: /mmba/ Disallow: /MSFTPSVC89/ Disallow: /Portfolio/ Disallow: /private/ Disallow: /public/ Disallow: /report/ #Allow: /SecurityNews/ #Allow: /site/ #Allow: /StEdwardTheConfessor/ #Allow: /tthc/ Disallow: /W3SVC88/ User-agent: sitecheck.internetseer.com Disallow: / User-agent: Hatena Antenna Disallow: / User-agent: NPBot Disallow: /