Bot Blocklist
ScraperBlock ships with a built-in list of 60+ user-agent signatures. A request is blocked when its user agent contains any entry (case-insensitive). You can see the list under Settings → ScraperBlock → Default Bot List.
AI Training Bots
| Bot | Organisation |
|---|---|
GPTBot | OpenAI |
Google-Extended | Google (AI training opt-out) |
CCBot | Common Crawl |
ClaudeBot, Claude-Web, anthropic-ai | Anthropic |
Bytespider, TikTokSpider | ByteDance / TikTok |
Amazonbot | Amazon |
Applebot-Extended | Apple (AI training opt-out) |
Meta-ExternalAgent, Meta-ExternalFetcher | Meta |
ImagesiftBot | Imagesift |
Diffbot | Diffbot |
AI2Bot | Allen Institute for AI |
YouBot | You.com |
cohere-ai, cohere-training-data-crawler | Cohere |
omgili | Webz.io |
AI Search & Assistant Bots
These bots fetch pages to answer users' questions and cite sources in AI search results, rather than to collect training data. They are blocked by default.
| Bot | Organisation |
|---|---|
OAI-SearchBot | OpenAI (ChatGPT search) |
ChatGPT-User | OpenAI (pages opened on a user's request) |
PerplexityBot | Perplexity AI |
Perplexity-User | Perplexity AI (pages opened on a user's request) |
MistralAI-User | Mistral AI (pages opened on a user's request) |
Turn on Allow AI Search Bots in Settings to let these bots read your pages so your site can appear in AI answers. AI training crawlers such as GPTBot and ClaudeBot stay blocked. The setting applies to runtime blocking, robots.txt directives and .htaccess rules.
SEO, Data & Commercial Crawlers
| Bot | Organisation |
|---|---|
SemrushBot | Semrush |
AhrefsBot | Ahrefs |
MJ12bot | Majestic |
DotBot | Moz |
BLEXBot | BLEXBot |
DataForSeoBot | DataForSEO |
MegaIndex | MegaIndex |
Seekport | Seekport |
ZoominfoBot | ZoomInfo |
TurnitinBot | Turnitin |
MeltwaterNews, Meltwater | Meltwater |
DomainStatsBot | DomainStats |
WellKnownBot | WellKnownBot |
Regional Search Engines & Archives
| Bot | Organisation |
|---|---|
YandexBot | Yandex |
Baiduspider | Baidu |
Sogou | Sogou |
PetalBot, PetalSearchBot | Huawei Petal Search |
Qwantify | Qwant |
Exabot | Exalead |
Neevabot | Neeva |
ia_archiver, archive.org_bot | Alexa / Internet Archive |
Link-Preview Bots
| Bot | Organisation |
|---|---|
facebookexternalhit | Facebook / Meta link previews |
LinkedInBot | LinkedIn link previews |
Slackbot-LinkExpanding | Slack link previews |
Discordbot | Discord link previews |
Blocking link-preview bots means shared links to your site show without a preview image or description on Facebook, LinkedIn, Slack and Discord. Use per-page control on pages you share often.
Automation & Headless Browsers
| Bot | Description |
|---|---|
Scrapy | Python scraping framework |
python-requests | Python HTTP library |
Go-http-client | Go HTTP library |
curl/ | cURL command line |
Wget/ | Wget command line |
HeadlessChrome | Headless Chrome browser |
PhantomJS | Headless browser (legacy) |
Puppeteer | Chrome automation library |
The full list is returned by ScraperBlock_Rules::get_bot_blocklist(). To block more bots, add their user-agent signatures in the Custom User-agents setting, one per line.
Common search crawlers - Googlebot, Bingbot, Slurp (Yahoo) and DuckDuckBot - are not on the list. Google-Extended is Google's AI training opt-out signal and is separate from Googlebot, so blocking it doesn't affect Google Search.