User-agent: Amazonbot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Applebot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: AwarioRssBot Disallow: / User-agent: AwarioSmartBot Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: cohere-ai Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: Diffbot Disallow: / User-agent: FacebookBot Disallow: / User-agent: FriendlyCrawler Disallow: / User-agent: GPTBot Disallow: / User-agent: img2dataset Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: Meltwater Disallow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-agent: peer39_crawler Disallow: / User-agent: peer39_crawler/1.0 Disallow: / User-agent: PerplexityBot Disallow: / User-agent: PiplBot Disallow: / User-agent: scoop.it Disallow: / User-agent: Seekr Disallow: / User-agent: YouBot Disallow: / User-agent: GeedoProductSearch Disallow: / User-agent: SeznamBot Disallow: / User-agent: ShopWiki Disallow: / User-agent: Slurp Disallow: / User-agent: Sogou web spider Disallow: / User-agent: spbot Disallow: / User-agent: SurveyBot Disallow: / User-agent: TweetmemeBot Disallow: / User-agent: Twitterbot Disallow: / User-agent: UnwindFetchor Disallow: / User-agent: Vagabondo Disallow: / User-agent: W3C_Validator Disallow: / User-agent: YandexBot Disallow: / User-agent: YisouSpider Disallow: / User-agent: ZyBorg Disallow: / # Allow all other user agents (e.g., Googlebot, Bingbot) User-agent: * Disallow: