User-Agent: * Disallow: /cgi-bin/ # e-seeds.jp (gsa-crawler) User-agent: gsa-crawler-jst Disallow: