User-agent: NetResearchServer User-agent: CJNetworkQuality User-agent: kulokobot User-agent: RoboCrawl User-agent: Gaisbot User-agent: Tutorial Crawler User-agent: IBM_Planetwide User-agent: Custo User-agent: Exabot User-agent: psbot User-agent: MJ12bot User-agent: IRLbot User-agent: Heritrix User-agent: ichiro Disallow: / User-agent: * Crawl-delay: 90 Disallow: /cgi-bin/tj-e.cgi Disallow: /cgi-bin/pro/ Disallow: /phpadsnew/ Disallow: /blog/page/1/ Disallow: /blog/category/discounts/page/1/ Disallow: /blog/category/museums/page/1/ Disallow: /blog/category/design/page/1/ Disallow: /im/alt/ Disallow: /ts2.8/ Disallow: /no.php