Why does OpenAI use web crawlers?
We use crawlers to validate the safety of web pages submitted as ads on ChatGPT. When you submit an ad, OpenAI may visit the landing page to ensure it complies with our policies. We may also use content from the landing page to determine when it’s most relevant to show the ad to users.
Which OpenAI crawlers should you allow?
You must allow OAI-AdsBot. We recommend allowing both OAI-AdsBot and OAI-SearchBot
OpenAI crawlers fail to crawl my website. What should I do?
Most websites have multiple layers of protection before a crawler can successfully access a webpage. We recommend working with your engineering/security team to validate that OpenAI crawlers can pass through each of the following layers:
1. robots.txt
Overview: The robots.txt file tells crawlers whether they are permitted to access certain parts of your website. OpenAI crawlers respect these rules. If access is disallowed in robots.txt, crawling will stop immediately.
Recommendation: Review your robots.txt configuration and confirm that OpenAI crawlers are explicitly allowed to access the relevant pages and paths.
User-agent: OAI-SearchBot
Allow: /
User-agent: OAI-AdsBot
Allow: /
2. Web Protection / Bot Mitigation
Overview: Many websites use services such as Cloudflare, Akamai, or other web protection providers to defend against DDoS attacks, scraping, and unauthorized traffic. These systems can mistakenly block legitimate crawlers, often returning 403 Forbidden errors. Because OpenAI crawlers can resemble automated traffic patterns, they may be denied unless specifically allowlisted.
Recommendation: Review your web protection or firewall configuration and allowlist OpenAI crawler traffic where possible, ideally based on our crawler user agents. Your engineering or infrastructure team should also inspect any automated bot mitigation rules that could be triggering false positives.
3. Human Verification / Anti-Bot Logic
Overview: Some websites implement additional application-level checks to verify that a visitor is human (for example: CAPTCHAs, JavaScript challenges, behavioral analysis, or session validation). Since OpenAI crawlers are automated systems, these checks may block access even if the crawler successfully passes earlier layers.
Recommendation: Review any human-verification or anti-automation logic implemented within your application and ensure OpenAI crawlers are exempted where appropriate, ideally by allowlisting our crawler user agents.
A Note on Stable IP Ranges
Some security systems require crawler traffic to originate from stable, publicly documented IP ranges before traffic can be reliably allowlisted.
Because crawler infrastructure may evolve over time, your engineering team should avoid relying solely on short-term IP observations from logs. Instead, we recommend validating traffic through a combination of: user-agent identification, verified bot programs (where supported), firewall allowlists, robots.txt behavior, and provider-level bot verification systems.
If you must allow a stable list of IP ranges, please reference:
A Note on Rate Limiting
Large batch uploads or sudden spikes in crawler traffic can sometimes trigger automated rate limiting or bot protection systems.
If you suspect rate limiting is occurring, ask your engineering team to review:
HTTP response codes (especially 429 Too Many Requests)
Firewall or CDN logs
Bot mitigation events
Request throttling rules
Traffic analytics around the time the crawler attempted access
This can help identify whether requests are being intentionally slowed or blocked by infrastructure protections.
You may also consider uploading ads across more time in smaller batches.
A Note on Cloudflare
OAI-AdsBot is now officially verified and allowlisted by Cloudflare.