AI access gates
An AI engine can't quote a page its crawler may not fetch. SiteUpward tests each AI search crawler separately.
- robots.txt rules for 16 AI and search user agents, including OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot
- CDN / WAF blocks and challenges (Cloudflare & co.)
- Content visible without JavaScript – most AI crawlers don't run it
- noindex, nosnippet and canonical traps