FAQ
Does blocking GPTBot affect Google Search ranking?
No. GPTBot is OpenAI's AI training crawler - it is separate from Googlebot (Google Search) and Google-Extended (Google's AI training opt-out). Blocking Google-Extended does not affect Search rankings.
Does robots.txt blocking actually stop AI bots?
Major AI companies (OpenAI, Anthropic, Google) have committed to respecting robots.txt directives. However, rogue scrapers ignore them. Use user-agent blocking or htaccess rules for enforcement.
Will rate limiting affect Google Search crawling?
Googlebot typically respects crawl delays and does not hammer pages rapidly enough to trigger a 60 requests/minute threshold. If you have aggressive crawling concerns, use Google Search Console to set a crawl rate limit directly.
Can I whitelist a bot?
Yes - use the scraperblock_bot_blocklist filter to remove a bot from the list, or the scraperblock_is_blocked filter to return false for a specific user-agent. See the Developer Reference.
Does ScraperBlock block legitimate search engine crawlers?
Only if they are explicitly on the blocklist. Common search crawlers (Googlebot, Bingbot, Slurp, DuckDuckBot) are NOT on the list. Semrush and Ahrefs bots are included as commercial scrapers - remove them if you use those tools for your own SEO.
Does it work with multisite?
The free version works on individual sites within a multisite network. Network-wide configuration is a Pro feature.
Is there a way to test if a bot would be blocked?
Go to Settings → ScraperBlock → Tools → Test User Agent and enter any user-agent string to see if it would be blocked.