Robots.txt Control
When Enable Robots.txt Blocking is on, ScraperBlock injects directives into WordPress's virtual robots.txt output via the robots_txt filter.
Example Output
The following is appended to your robots.txt:
# ScraperBlock
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
...One User-agent / Disallow: / pair is added for each bot on the built-in list. Custom user agents are blocked at runtime but aren't added to robots.txt.
When Allow AI Search Bots is on, ScraperBlock leaves AI search bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, MistralAI-User) out of these directives. See Bot Blocklist.
Robots.txt directives are advisory only - compliant bots respect them, but malicious scrapers do not. Use htaccess blocking or WordPress-level blocking for enforcement.
ScraperBlock adds its rules through WordPress's virtual robots.txt. SEO plugins that edit the virtual file (Yoast SEO, Rank Math) keep working - ScraperBlock's rules are added after theirs. If a physical robots.txt file exists in your site's root folder, the web server serves that file instead and ScraperBlock's rules won't appear.