Attackers love to pretend to be Googlebot, because many firewalls wave search engines through. NOC checks every crawler claim against the IP addresses the search engine publishes, and every "browser" against how real browsers behave. Impostors are stopped at the edge.
Any program can send a request that says it's Googlebot. Scanners do it on purpose: some sites and firewalls trust search engines, rate limits often skip them, and nobody wants to block Google by mistake. In one night, a single NOC server saw thousands of requests from fake crawlers. One IP cycled through Googlebot, Bingbot and OpenAI's crawler while looking for .env files with passwords.
The real crawlers come from IP ranges their owners publish, so a "Googlebot" from a cheap VPS or a home connection is an impostor. NOC downloads those lists every day and checks each claim.
A request claiming to be Googlebot, Bingbot, DuckDuckBot, Applebot, an OpenAI crawler or Meta's crawler must come from that company's network. If it doesn't, it's blocked. This is on for every site.
Requests that claim to be a browser but don't behave like one, and known automation tools, get a CAPTCHA.
Verified good bots are let through, and a real person who gets the challenge passes it in one click.
Fake crawler detection is always on. Fake browser detection is in the Security tab of each site.
Set fake browser detection for any site.
curl "https://my.noc.org/api?apikey=YOUR_API_KEY&action=cdn/set/fake_browser&website=example.com&value=site"
curl "https://my.noc.org/api?apikey=YOUR_API_KEY&action=cdn/set/fake_browser_action&website=example.com&value=captcha_invisible"
Learn the basics: What is a bot? · Blocking user agents
Get access to our full suite of services, including DNS, WAF/CDN and Monitoring, or use only what you need. Our WAF/CDN operates on a distributed anycast network with over 20 data centers, delivering exceptional global performance.
Free 14-Day Trial