AI companies crawl the web to train their models and power their answers. Some respect robots.txt, many don't, and some hit your server harder than any search engine. NOC lets you block them at the edge, choose which ones to allow, and keep your search traffic.
AI crawlers copy your articles, product descriptions and documentation to train models that may then answer your visitors' questions without sending them to you. Some of them also crawl aggressively, adding load to your server and costs to your hosting.
robots.txt is a request, not a rule, and you have to know every crawler's name to use it. NOC enforces your choice at the edge: crawlers you block get a response from our network and never reach your server.
Googlebot and Bingbot keep crawling and ranking your site as before.
Links shared on Slack, WhatsApp, Facebook or X still show their preview.
Always served, so well-behaved crawlers can read your preferences.
Turn it on for one site or a thousand.
curl "https://my.noc.org/api?apikey=YOUR_API_KEY&action=cdn/aicrawlers/set&website=example.com&enabled=true&urls=/&allow=openai,anthropic"
curl "https://my.noc.org/api?apikey=YOUR_API_KEY&action=cdn/aicrawlers/set&website=example.com&enabled=true&mode=training"
Google-Extended and Applebot-Extended to your robots.txt; the dashboard shows the exact lines.Learn the basics: What is a bot? · Blocking user agents
Get access to our full suite of services, including DNS, WAF/CDN and Monitoring, or use only what you need. Our WAF/CDN operates on a distributed anycast network with over 20 data centers, delivering exceptional global performance.
Free 14-Day Trial