AI Crawler Blocking

Your content, your call on AI

AI companies crawl the web to train their models and power their answers. Some respect robots.txt, many don't, and some hit your server harder than any search engine. NOC lets you block them at the edge, choose which ones to allow, and keep your search traffic.

Free 14-Day Trial
No credit card required. Included for every site.
Blocks GPTBot, ClaudeBot, Bytespider, CCBot and many more
Block AI training only, or every AI crawler
Allow the AI vendors you choose
Googlebot, Bing and link previews are never affected

What AI crawlers take

AI crawlers copy your articles, product descriptions and documentation to train models that may then answer your visitors' questions without sending them to you. Some of them also crawl aggressively, adding load to your server and costs to your hosting.

robots.txt is a request, not a rule, and you have to know every crawler's name to use it. NOC enforces your choice at the edge: crawlers you block get a response from our network and never reach your server.

What stays untouched

Search engines

Googlebot and Bingbot keep crawling and ranking your site as before.

Link previews

Links shared on Slack, WhatsApp, Facebook or X still show their preview.

robots.txt

Always served, so well-behaved crawlers can read your preferences.

Your options

What to block
AI training only blocks the crawlers that collect data to train models, and lets AI search and on-demand requests through. All AI crawlers blocks both.
Vendors to allow
Always let through the ones you want: OpenAI, Anthropic, Google, Perplexity, Meta, Amazon, Mistral, xAI, DuckDuckGo or You.com.
Which pages
The whole site, or only some paths, like your blog or documentation. /robots.txt is always allowed, so crawlers can still read your rules.
Response
A block page, or a stealth response (an empty page, or a generic 403, 404 or 503) so scrapers don't know they were stopped.

Automate it with the API

Turn it on for one site or a thousand.

Block AI crawlers on the whole site, except OpenAI and Anthropic
curl "https://my.noc.org/api?apikey=YOUR_API_KEY&action=cdn/aicrawlers/set&website=example.com&enabled=true&urls=/&allow=openai,anthropic"
Block AI training crawlers only
curl "https://my.noc.org/api?apikey=YOUR_API_KEY&action=cdn/aicrawlers/set&website=example.com&enabled=true&mode=training"

Full API reference →

Frequently asked questions

No. Googlebot and Bingbot aren't affected. Note that Google trains Gemini and Apple trains Apple Intelligence on what Googlebot and Applebot fetch for search, so those can't be blocked without leaving search. To opt out of their AI training, add Google-Extended and Applebot-Extended to your robots.txt; the dashboard shows the exact lines.

Crawlers that hide behind a browser user agent are caught by other features: fake crawler detection and datacenter blocking, since most scrapers run on cloud servers.

Yes. Our team adds new AI crawlers as they appear. It's part of the managed rules every site gets.

Learn the basics: What is a bot? · Blocking user agents

Related features

Security that works, at a price that makes sense

$5per domain per month.

Get access to our full suite of services, including DNS, WAF/CDN and Monitoring, or use only what you need. Our WAF/CDN operates on a distributed anycast network with over 20 data centers, delivering exceptional global performance.

Free 14-Day Trial