Fake Crawler Detection

Fake Googlebots, caught before they reach your site

Attackers love to pretend to be Googlebot, because many firewalls wave search engines through. NOC checks every crawler claim against the IP addresses the search engine publishes, and every "browser" against how real browsers behave. Impostors are stopped at the edge.

Free 14-Day Trial
No credit card required. Included for every site.
Google, Bing, DuckDuckGo, Apple, OpenAI and Meta crawlers verified
Checked against each company's own published IP lists, refreshed daily
Scripts pretending to be Chrome or Safari get a challenge

A night of fake Googlebots

Any program can send a request that says it's Googlebot. Scanners do it on purpose: some sites and firewalls trust search engines, rate limits often skip them, and nobody wants to block Google by mistake. In one night, a single NOC server saw thousands of requests from fake crawlers. One IP cycled through Googlebot, Bingbot and OpenAI's crawler while looking for .env files with passwords.

The real crawlers come from IP ranges their owners publish, so a "Googlebot" from a cheap VPS or a home connection is an impostor. NOC downloads those lists every day and checks each claim.

Two checks, working together

Fake crawlers

A request claiming to be Googlebot, Bingbot, DuckDuckBot, Applebot, an OpenAI crawler or Meta's crawler must come from that company's network. If it doesn't, it's blocked. This is on for every site.

Fake browsers

Requests that claim to be a browser but don't behave like one, and known automation tools, get a CAPTCHA.

Real people first

Verified good bots are let through, and a real person who gets the challenge passes it in one click.

Where it applies

Fake crawler detection is always on. Fake browser detection is in the Security tab of each site.

Where to check browsers
Login and signup pages (the default for every site), the whole site, or off.
Action
CAPTCHA (the default), invisible CAPTCHA, or block.

Automate it with the API

Set fake browser detection for any site.

Check browsers on the whole site, with an invisible CAPTCHA
curl "https://my.noc.org/api?apikey=YOUR_API_KEY&action=cdn/set/fake_browser&website=example.com&value=site"
curl "https://my.noc.org/api?apikey=YOUR_API_KEY&action=cdn/set/fake_browser_action&website=example.com&value=captcha_invisible"

Full API reference →

Frequently asked questions

Not on a site that points straight to NOC: real Googlebot comes from Google's published ranges and is let through. If another CDN like Cloudflare sits in front of NOC, turn on Using a third-party CDN in the General tab, so we see the crawler's real IP instead of the CDN's.

Google, Bing, DuckDuckGo, Apple, OpenAI and Meta, using the IP lists each of them publishes. We add more as they publish their ranges.

A request that says it's a browser but doesn't act like one, or carries the signature of a known automation tool.

Learn the basics: What is a bot? · Blocking user agents

Related features

Security that works, at a price that makes sense

$5per domain per month.

Get access to our full suite of services, including DNS, WAF/CDN and Monitoring, or use only what you need. Our WAF/CDN operates on a distributed anycast network with over 20 data centers, delivering exceptional global performance.

Free 14-Day Trial