GPTBot crawl frequency: how often do AI crawlers actually read your website?
There is no fixed GPTBot crawl frequency published by OpenAI. Its crawler documentation describes four separate bots, and none of them comes with a “visits every X days” promise. The only timing it gives is roughly 24 hours for a robots.txt change to show up in search results. The second piece of news comes from Cloudflare: from 15 September 2026, on newly connected domains and Free plan sites, training and agent bots are blocked by default on pages that show ads.
For a business owner, the takeaway is simple: you cannot know when an AI crawler will walk into your shop, but you decide what it finds when it does. Your server log shows which bot comes and how often. A page that is error-free, readable in plain HTML and updated on a schedule is the strongest answer to that uncertainty.
Which OpenAI crawlers exist, and what does each one do?
BU BÖLÜMÜN ÖZETİ
- GPTBot and OAI-SearchBot are not the same
- The IP lists are public
- Robots.txt takes about 24 hours
OpenAI’s documentation lists four crawlers, each with its own job. GPTBot collects content to train models. OAI-SearchBot crawls so your site can appear in ChatGPT search answers. ChatGPT-User opens a page on the spot when a user asks ChatGPT something. OAI-AdsBot checks pages submitted as ads on ChatGPT. Which door you open and which you close changes the outcome directly.
| Crawler | Job | What happens if you block it? |
|---|---|---|
| GPTBot | Collects content for model training | Signals your content should not be used for training |
| OAI-SearchBot | Visibility in ChatGPT search | According to the docs, you will not appear in search answers |
| ChatGPT-User | Opens pages at a user’s request | Does not crawl automatically; robots.txt rules may not apply |
| OAI-AdsBot | Safety checks on ad landing pages | Its data is not used for model training |
GPTBot and OAI-SearchBot are not the same
Many owners think blocking GPTBot takes them out of ChatGPT, but search visibility is decided by OAI-SearchBot. Managing the two on separate lines is like pulling the shutter halfway down: closed to training, open to search.
The IP lists are public
OpenAI publishes each crawler’s IP addresses in a JSON file. Use them to check whether a “GPTBot” visit in your log really comes from OpenAI, because fake bots exist.
Robots.txt takes about 24 hours
According to the docs, a robots.txt change takes about 24 hours to reach search results. If you wonder why you are not being recommended, see the guide why isn’t my website recommended by AI.
What is officially known about crawl frequency?
What is officially known is that the frequency is not fixed. Neither OpenAI nor Google commits to how often its crawler visits a site; crawling varies with site size, update pace and server capacity. Your firm measures are your own server log and the crawl reports in search dashboards.
Your server log gives you the real calendar
The access log in your hosting panel records the time and user-agent of every visit. Search it for “GPTBot”, “OAI-SearchBot”, “ClaudeBot” and “PerplexityBot” and you see your site’s real crawl calendar.
Search dashboards show crawling too
The Crawl Stats report in Google Search Console shows Googlebot’s visits. Bing Webmaster Tools offers a similar crawl report. The step-by-step guide Bing Webmaster Tools setup gets you to that report.
What do the numbers say about how much AI bots crawl?
The scale is large and wasteful. According to Vercel’s December 2024 network analysis, GPTBot makes 569 million requests a month and Claude’s crawler 370 million. Googlebot leads the same network with 4.5 billion. The four big AI crawlers combined come to a little over 28 percent of Googlebot’s volume.
A third of requests are wasted
The same analysis finds 34.82 percent of ChatGPT’s crawler requests hit 404 pages and 14.36 percent hit redirects. Googlebot’s 404 rate is 8.22 percent. A site full of broken links throws away a bot’s already limited visit.
JavaScript is not executed
The analysis shows major AI crawlers fetch JavaScript files but do not execute them. If your price list or service cards load through JavaScript after the page opens, the bot sees an empty shop window.
What changes at Cloudflare on 15 September 2026?
Instead of blocking AI bots as one group, Cloudflare now splits them into three classes: search, agent and training. From 15 September 2026, on newly connected domains, new customers and Free plan sites, training and agent bots are blocked by default on pages that show ads. Search bots stay allowed. Existing paid customers are not changed automatically.
Multi-purpose bots get the strictest rule
Some crawlers do both search and training, and reports say the strictest rule applies to them. Blocking training may also block search.
Check your own setting
If your site sits behind Cloudflare, open the AI bots section and note which class is open. When customers are not walking through the door, sometimes the problem is the door itself.
What it means for you: not when the crawler comes, but what it finds
You cannot set the crawl frequency, but you can set the yield of every visit. A site with few broken links, plain HTML content, a correct robots.txt and regular updates gets more from each visit. This is the first stone of the groundwork that lets your name appear in AI answers.
Write robots.txt on purpose
Manage training, search and user bots on separate lines, and check the file against what you meant to allow.
llms.txt is a note on the counter
llms.txt is a text file that offers AI crawlers a summary of your site. It does not raise crawl frequency, but it guides the bot that arrives. A plain explanation is in what is llms.txt.
Update calendar: which page needs attention, and how often?
Whenever the bot arrives, it should meet current information, and a fixed maintenance calendar is the easiest way to get there. The calendar below splits weekly, monthly and quarterly jobs for a small business site.
| How often | Task | Time |
|---|---|---|
| Weekly | Check prices, opening hours, stock and offers | 15 minutes |
| Monthly | Review bot visits in the server log, fix 404 pages | 45 minutes |
| Monthly | Add one new answer block for a common question | 1 hour |
| Quarterly | Review robots.txt, sitemap and llms.txt | 1 hour |
| Quarterly | Update dates and figures on service pages | 2 hours |
| Every 6 months | Report which bots visit and which do not | 1 hour |
The price at the counter and online should match
If a service is TRY 450 in the shop but TRY 350 online, an AI answer may carry the old figure. Update the site the same day.
How are crawler settings handled in the AI SEO campaign?
BU BÖLÜMÜN ÖZETİ
- Diagnosis first, then settings
- 108 articles give bots a reason to read
- Measured groundwork, not a guarantee
In the AI SEO campaign, crawler settings are among the first jobs of the setup. The package is TRY 36,000 + VAT at a fixed price, delivered in 30 days; with 20 percent VAT that is TRY 43,200, but check the current rate on your invoice. The scope covers llms.txt and AI crawler settings, structured data, Bing Webmaster Tools and Search Console setup, and AI traffic measurement in GA4. A diagnosis card measures the situation at the start and at the end.
Diagnosis first, then settings
You share your site and target questions, a diagnosis card is prepared, then the turnkey setup begins. Blocked bots and 404 pages show up at this stage.
108 articles give bots a reason to read
The campaign writes 108 articles across 6 question groups, each answering a specific question. The full logic is explained in AI SEO with 108 articles. Only 12 businesses are taken per period; to apply, visit the AI SEO campaign page.
Measured groundwork, not a guarantee
The campaign does not guarantee being recommended in AI answers. It builds and measures the groundwork that makes the site readable and recognisable.
Quick Summary
- OpenAI does not publish a fixed crawl frequency for GPTBot.
- A robots.txt change reaches search results in about 24 hours.
- GPTBot crawls for training, OAI-SearchBot for ChatGPT search.
- AI crawlers do not execute JavaScript, so content should be in plain HTML.
- From 15 September 2026, Cloudflare blocks training and agent bots by default on some sites.
- Your real crawl calendar is in your server log.
Short Glossary
Crawler: A crawler is software used to visit web pages automatically and read their content.
Robots.txt: Robots.txt is a text file used to tell bots which parts of a site they may enter.
Server log: A server log is a file used to record the time and identity of every visit to a site.
Next Step
If you want crawlers to find the right information when they visit, get the setup delivered in 30 days. Apply for the AI SEO campaign and start with your site’s diagnosis card.
Sık Sorulan Sorular
According to OpenAI’s documentation, GPTBot crawls for model training. Visibility in ChatGPT search is decided by OAI-SearchBot. You can block GPTBot and still allow OAI-SearchBot.
According to OpenAI’s documentation, a robots.txt update takes about 24 hours to reach search results. Wait a day after the change, then check.
In the AI SEO campaign, the setup is delivered in 30 days. It starts with a diagnosis card and ends with a final diagnosis card and a handover video.
Updated: October 2026