FAQ
Does blocking GPTBot affect Google Search ranking?
No. GPTBot is OpenAI's AI training crawler - it is separate from Googlebot (Google Search) and Google-Extended (Google's AI training opt-out). Blocking Google-Extended does not affect Search rankings.
Does robots.txt blocking actually stop AI bots?
Major AI companies (OpenAI, Anthropic, Google) have committed to respecting robots.txt directives. However, rogue scrapers ignore them. Use user-agent blocking or htaccess rules for enforcement.
Will rate limiting affect Google Search crawling?
No. Googlebot isn't on the blocklist, and search engines spread their crawling over many IP addresses, so one crawler IP stays far below the default limit of 60 page requests a minute. If you ever see rate_limit entries for a search engine in the log, raise Requests per minute.
Can I whitelist a bot?
You can allow AI search and assistant bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, MistralAI-User) by turning on Allow AI Search Bots in Settings. Other built-in bots can't be removed from the list individually. Developers can use the scraperblock_bypass_request filter to let specific requests skip runtime blocking - see the Developer Reference. That filter does not change the robots.txt or .htaccess rules.
Will blocking AI bots stop ChatGPT from recommending my site?
Not if you turn on Allow AI Search Bots. AI training crawlers such as GPTBot and CCBot stay blocked, while AI search bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User and MistralAI-User) can read your pages so your content can appear in AI answers. The setting is off by default, so these bots are blocked until you turn it on.
Does ScraperBlock block legitimate search engine crawlers?
Googlebot, Bingbot, Slurp (Yahoo) and DuckDuckBot are not on the list, so your site stays indexed in those search engines. The list does include some regional search crawlers (YandexBot, Baiduspider, PetalSearchBot, Sogou, Qwantify), SEO tools (SemrushBot, AhrefsBot), and link-preview bots (facebookexternalhit, LinkedInBot, Slackbot-LinkExpanding, Discordbot). Built-in entries can't be removed individually - if you rely on one of them, use per-page control for the pages concerned.
Does it work with multisite?
The free version works on individual sites within a multisite network. Network-wide configuration is a Pro feature.
Is there a way to test if a bot would be blocked?
There's no built-in tester. Check whether any entry in the Default Bot List or your Custom User-agents appears inside the user-agent string (matching is case-insensitive). You can also request a page with that user agent - for example curl -A "GPTBot" https://yoursite.com/ - and check for a 403 response and a new entry on the Logs tab.