Robots.txt Control
When Enable Robots.txt Blocking is on, ScraperBlock injects directives into WordPress's virtual robots.txt output via the robots_txt filter.
Example Output
The following is appended to your robots.txt:
# ScraperBlock - AI Bot Restrictions
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
...⚠️
Robots.txt directives are advisory only - compliant bots respect them, but malicious scrapers do not. Use htaccess blocking or WordPress-level blocking for enforcement.
If you use a dedicated SEO plugin (Yoast SEO, Rank Math) that manages robots.txt, ScraperBlock's directives are appended after the SEO plugin's output to avoid conflicts.