Bot protection and catalog - IP Location Block

Advanced mode Start with Simple mode

Bot rules control crawler behavior separately from visitor location rules. Open Advanced > Front-end target settings > Bot protection.

Start with maintained presets

Recommended bot rule presets — Presets group search, feed, social, AI, and scraper policies into clear choices.

Choose the behaviors your site needs, such as allowing verified search engines, blocking AI training crawlers, or blocking aggressive SEO scrapers.

Understand allow and block rules

Block rules can match a User-Agent token directly. Search-engine allow rules should use verified host checks so a visitor cannot pass by copying a crawler name.

Reverse DNS verification

Turn on Reverse DNS lookup when HOST allow rules must verify search engines. Without it, those allow rules pass on the User-Agent token alone.

Test a User-Agent

Advanced User-Agent rule builder — Each rule combines an allow or block action with a country, host, feed, ASN, referrer, or IP qualifier.

Paste a complete User-Agent into the tester to see which saved rule matches and whether the result is allow or block. This preview does not make a live request.

Advanced User-Agent tester — The tester previews the first saved rule that matches a complete User-Agent string.

Save and monitor

Save Changes, then use Validation logs to confirm crawler decisions. Review the rules periodically because crawler names and desired policies change.