Chapter 9: How AI Bots See Your Site

Last updated: November 1, 2026.

Say a site owner adds “Disallow: /” for GPTBot to robots.txt and considers the job done. Looking at the logs, they might find three different things: GPTBot really has stopped coming, ChatGPT-User keeps coming, and requests that call themselves GPTBot but don’t come from OpenAI’s IP ranges are still there. Each of those tells a different story.

Chapter 2 introduced AI bots: which company runs which bot, what robots.txt controls, how a CDN like Cloudflare steps in before robots.txt, and why important information needs to be in the initial HTML. This chapter goes one level deeper. Which bot should be allowed to do what, what do bots actually do on your site, and is the bot really the bot it claims to be? The answers are in your server records, the logs. At the end we look at a much-discussed file, llms.txt.

1. Three kinds of bots

AI companies’ bots do three different jobs. Some collect content to train future models, some crawl and index pages for the AI’s search feature, and some open a page on the spot when a user asks a question. Let’s expand the table from Chapter 2 a little:

Perplexity says PerplexityBot doesn’t collect content to train AI models and is used to show and link to sites in Perplexity’s search results (Perplexity). Apple writes that Applebot-Extended doesn’t crawl pages; it only determines whether data collected by Applebot may be used to train Apple’s generative AI models, and pages that block it keep appearing in search results (Apple). It’s the same logic as Google-Extended.

In March 2026 Google added Google-Agent to its list of user-triggered fetchers. It identifies AI agents running on Google infrastructure that browse the web and take actions at a user’s request (PPC Land). OpenAI’s list also includes OAI-AdsBot, which only checks pages submitted as ads in ChatGPT; OpenAI states that the data it collects isn’t used to train models (OpenAI).

The three groups carry very different weight in traffic. According to data Cloudflare published in August 2025, over the 12 months to July 2025, 80% of AI bot crawling was for training, 18% for search and 2% for user actions (Cloudflare). The AI traffic you’ll see most in your logs is probably training traffic, which doesn’t send you visitors directly.

So instead of one decision for all three groups, you need three separate ones.

2. Allowing training bots, or not

The content training bots collect is the path into the model memory we described in Chapter 3. Blocking these bots means you don’t want your content used to train future models. OpenAI defines it exactly that way: blocking GPTBot indicates that your site’s content shouldn’t be used to train generative AI models (OpenAI).

The most common objection to training bots is that they crawl your site and send nobody back. There’s data behind that objection. Cloudflare compares the number of pages a platform crawls with the number of visitors it refers. In July 2025, for every referred visit there were about 38,066 crawls from Anthropic, about 1,091 from OpenAI and about 195 from Perplexity (Cloudflare). These figures cover all bots together and are limited to Cloudflare’s network, but the direction is clear: crawling for training doesn’t bring you traffic in the short term.

Still, I don’t think the training bot decision should come down to a traffic calculation. What training gives you isn’t clicks, it’s memory. If a model learned about your brand, products and expertise from its training data, it can mention you even in questions where it doesn’t search. That value is hard to measure, but ignoring it is a mistake.

It’s better to decide based on your business model. Manufacturers, e-commerce, B2B and service companies sell products, not content; having models recognize their brand correctly works in their favor, so in most cases it makes sense to allow training bots. For publishers and anyone who makes money from content, the content is the product. Blocking training bots, or keeping the door shut to negotiate a license, is a defensible choice for them. Those in between can open their blog and guides to training while closing off paid reports or member areas; robots.txt lets you write rules by directory.

Whatever you decide, keep this distinction straight: blocking a training bot doesn’t take you out of AI search; blocking a search bot does.

3. Search and user bots

With search bots, the picture is simpler. OAI-SearchBot, Claude-SearchBot and PerplexityBot are what let your site be cited in those platforms’ search answers. Blocking them lowers your chance of being cited on that platform, or wipes it out. Google has no separate search bot; as we saw in Chapter 2, the rule for AI Overviews and AI Mode is the Googlebot rule.

robots.txt changes don’t take effect instantly. OpenAI says it can take about 24 hours for a robots.txt update to reach its systems, and Perplexity gives the same timeframe. Google generally caches robots.txt for up to 24 hours (Google Search Central).

User bots are a separate category, and an often misunderstood one. They don’t crawl the web automatically. They kick in when a user gives ChatGPT a link to your page and says “summarize this,” or when Perplexity opens your page on the spot to answer a question. Platforms also treat robots.txt differently for these bots. For ChatGPT-User, OpenAI says, “Because these actions are initiated by a user, robots.txt rules may not apply.” Perplexity writes that because a user initiates the fetch, Perplexity-User generally ignores robots.txt rules. Google-Agent is also in the user-triggered fetcher class, and robots.txt rules may not block it.

That has two consequences. Don’t assume these bots will stay away just because you wrote “Disallow” in robots.txt; if you want to block them, you need a rule at the server or CDN level. And second, in my view, for most businesses blocking these bots means shooting themselves in the foot. A user bot visiting your site means a person is asking something about you right now; you’re shutting the door on someone who has already found you.

Google’s John Mueller said something similar in June 2026: he said the most basic optimization for agents is “not blocking agents” and that he thinks this will be the biggest obstacle for most sites (Search Engine Journal).

A newer development is bots proving their identity cryptographically. With a method called Web Bot Auth, the bot digitally signs its request and the site can verify the signature. Cloudflare writes that ChatGPT Agent signs its requests this way (Cloudflare). Google is still testing the protocol for Google-Agent. We cover agents and commerce separately in Chapter 15.

4. Writing robots.txt

Once you’ve decided, you have to write the decision into the file correctly, and there are a few traps.

The one most often missed is that the most specific group wins. If robots.txt has more than one “User-agent” group, Google’s crawlers use the most specific group matching their name and ignore the others (Google Search Central). Under the robots.txt standard, too, a bot that finds a group with its own name doesn’t use the wildcard group. So if your file blocks /sepet/ (cart) and /hesabim/ (my account) under “User-agent: *” and you’ve opened a separate group for GPTBot, GPTBot no longer reads the wildcard rules. You have to write those rules into the GPTBot group as well.

Google-Extended and Applebot-Extended aren’t bots but control tokens; in robots.txt, though, they’re written the same way, as a User-agent line.

Crawl-delay doesn’t apply to everyone. Anthropic says it supports the non-standard Crawl-delay rule (Anthropic); Google and Applebot don’t.

If there’s no rule for a bot, it may follow another bot’s rules. Apple writes that if robots.txt has no rule for Applebot but does have one for Googlebot, Applebot follows the Googlebot rules.

The file itself shouldn’t return an error either. When the robots.txt URL returns a 5xx error, Google stops crawling the site for the first 12 hours; it treats 4xx errors other than 429 as if there were no robots.txt at all.

A file that puts these rules together, closing training bots while keeping search and user bots open, could look like this. It’s an example; adapt it to your own decision:

User-agent: *
Disallow: /sepet/
Disallow: /hesabim/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /sepet/
Disallow: /hesabim/

Sitemap: https://www.ornek.com/sitemap.xml

The /sepet/ and /hesabim/ rules are repeated in the search bots’ group. Without them, the most-specific-group rule would let those bots into those directories too.

After writing the file, apply the rule from Chapter 2: read the live robots.txt in a browser at site.com/robots.txt. Your CDN may be adding lines you didn’t write to the top of the file.

5. Testing what the bot sees

Chapter 2 mentioned Vercel’s analysis from late 2024: GPTBot, ClaudeBot and PerplexityBot downloaded JavaScript files but didn’t execute them. The simplest way to test this on your own site is to read the page’s raw HTML, which is what “View page source” does in a browser. A slightly more technical way is to request the page from the command line with a bot’s identity:

curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://www.ornek.com/urun/ | grep -i "kapasite"

This command downloads the page without running JavaScript and searches it for the word “kapasite” (capacity). If the word doesn’t come up, the capacity information is being loaded with JavaScript.

Check three things in this test. Are price, stock, technical specs, the product description, FAQ answers and contact details in the raw HTML? Are tabs and expandable sections filled in, or is the table that loads when you click the “Technical specs” tab empty in the raw HTML? And does the server return the same response to the bot? Some servers and security plugins return a different page, a verification screen or a 403 error when they see a bot identity.

A caveat on that last point. This test imitates the bot’s identity but doesn’t come from the bot’s IP address. As we saw in Chapter 2, a CDN like Cloudflare recognizes verified bots by IP address and signature, so your test may get a different result from the real bot. Compare the test result with your logs; the real proof is the status code the actual bot received.

Take a filling machine manufacturer whose product page keeps capacity, suitable product types and nozzle options in a “Technical Specs” tab that’s filled in with JavaScript. Google may see that information. A bot that doesn’t run JavaScript may never see “suitable for foaming products.” In the user’s question from Chapter 1, that’s exactly the information that decides whether you make the list.

6. Log analysis

robots.txt says what you want; logs say what happened. In my view, any discussion about AI bots should start with the logs. In Chapter 2 I said server logs are the only real source for how often each bot visits; here’s how to read them.

A server log is a file that records every request to your site as one line. You can get logs from your hosting panel, your server or, if you use a CDN, the CDN’s logging service. On a site behind a CDN, some requests are answered at the CDN and never reach the server, so to see blocked bots you also need to look at the CDN’s records.

In the common log format, a single line looks like this (the IP address is an example address used in documentation):

203.0.113.10 - - [01/Nov/2026:10:15:32 +0300] "GET /dolum-makinesi/kopuren-urunler/ HTTP/1.1" 200 48211 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"

Five pieces of this line matter here. The IP address is where the request came from, and it’s how we verify whether the bot is real. The date and time tell you when the bot came, and the requested URL tells you which page was crawled. The status code gives the result: 200 success, 301 redirect, 404 not found, 403 blocked, 5xx server error. The user-agent at the end is the name the bot gives itself.

A log file can have millions of lines. You don’t need an expensive tool to start; a few simple commands produce the first table. To see how many requests each AI bot made:

grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-Agent" access.log | sort | uniq -c | sort -rn

To see the status codes a bot received:

grep "OAI-SearchBot" access.log | awk '{print $9}' | sort | uniq -c

To see which pages a bot visits most:

grep "ChatGPT-User" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

These commands help answer:

• Which bots are coming? If a search bot you allow in robots.txt never shows up, a layer in between may be blocking it.
• What status codes are they getting? If search bot requests show a high rate of 403 or 5xx, the bot is arriving and being turned away.
• Are they reaching your important pages? If bots spend their time on filter URLs, old campaign pages or 404s, the crawl budget problem from Chapter 2 applies to AI bots too.
• Are they reading robots.txt? If the bot’s robots.txt request doesn’t appear in the logs, or it gets an error, your rules may not be applied.
• How quickly are new pages crawled? Check when a search bot first visited a page you published.

Pay separate attention to user bot records. User bot logs are the trail of real human demand. If ChatGPT-User, Claude-User or Perplexity-User hit one of your pages, someone just asked about you on that platform and the platform opened your page to answer. These requests usually don’t appear as visits in your analytics tool, but they stay in the logs. Knowing which pages user bots open most is one of the rare pieces of data that shows what topics people turn to you for in AI. You can add it as a supplementary line to the business results report from Chapter 8.

On large sites, doing this from the command line gets hard. Then SEO tools that do log analysis, or loading the logs into a data warehouse and querying them, are more practical. The logic is the same: bot, page, status code, time.

7. Weeding out fake bots

Anyone can write a user-agent. A moment ago we introduced ourselves as OAI-SearchBot with a single command. Attackers and scrapers do the same, posing as Googlebot or GPTBot to get past blocks. Counting bot names in your logs without verifying them will mislead you.

One way to verify is a reverse DNS check. Google describes it in four steps (Google Search Central): run a reverse DNS lookup on the IP address from the log, check that the resulting domain is googlebot.com, google.com or googleusercontent.com, then run a forward DNS lookup on that domain and confirm the resulting IP matches the one in the log. On the command line it looks like this:

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

The first command finds the domain from the IP, the second finds the IP from the domain. If they match, the bot is real. Apple recommends the same method for Applebot; genuine Applebot requests resolve to the applebot.apple.com domain.

The other way is published IP lists. Most AI companies publish their bots’ IP ranges as JSON files:

• OpenAI: gptbot.json, searchbot.json, chatgpt-user.json and adsbot.json, on the openai.com domain (OpenAI).
• Anthropic: claude.com/crawling/bots.json (Anthropic).
• Perplexity: perplexitybot.json and perplexity-user.json, on the perplexity.com domain (Perplexity).
• Google: common-crawlers.json for Googlebot and other common crawlers, user-triggered-fetchers.json for user-triggered fetchers, and user-triggered-agents.json for agents. Google last updated this page on March 20, 2026.

These lists change from time to time; if you’re verifying, pull them from the source regularly instead of copying them by hand. Anthropic adds a warning: if you block by IP address, you may also stop bots from reading your robots.txt. If you want a block, use robots.txt first.

The best-known incident showing why verification matters happened in August 2025. Cloudflare claimed that Perplexity was sending requests to sites that had blocked it in robots.txt, posing as the Chrome browser and coming from addresses outside Perplexity’s published IP ranges. According to Cloudflare’s data, the declared Perplexity-User made 20–25 million requests a day while this undeclared crawler made 3–6 million (Cloudflare). Perplexity denied the claims and called Cloudflare’s post a “sales pitch” (TechRepublic).

Whoever is right, the incident has a lesson. A bot name in your logs is a claim, not proof. Count verified and unverified requests separately and you’ll both filter out fake traffic and see how real bots actually behave.

8. Server load: slow bots down instead of blocking them

Some sites block AI bots not over content concerns but because of server load. On a small server, heavy crawling can slow the site down, and the concern is understandable. But in my view a load problem is a capacity issue, not an access decision. Writing “Disallow: /” in robots.txt because of load is trying to fix the problem in the wrong place.

Measure with the logs first: which bot is creating the load, how many requests an hour it makes, and which URLs it goes to? Often the problem isn’t the bot but the URLs the site generates. If, on an e-commerce site that generates thousands of URLs through filter, sort and parameter combinations, bots spend most of their time on those URLs, closing them off in robots.txt reduces both load and wasted crawling.

If a bot still makes too many requests, you have a few tools:

• Crawl-delay: Anthropic says it supports this rule, so you can set a wait time between requests for ClaudeBot. Google and Applebot ignore it.
• Temporary slowdown: for emergencies, Google recommends returning 500, 503 or 429 instead of 200 to crawl requests; a Retry-After header can be sent with 503 and 429. This is a fix for a few hours or a day or two. If it goes on for several days on the same URL, the URL can drop out of the index, and because the slowdown applies to the whole domain, discovery of new pages and price and stock updates are delayed too (Google Search Central).
• Rate limiting at the CDN: capping requests above a certain rate instead of blocking the bot entirely. When you do, make sure verified search bots and Googlebot are excluded from the rule.
• Caching: serving pages to bots from cache cuts server load considerably.

Watch for a side effect with caching. On the news site from Chapter 2, the sitemap was cached for about 10 minutes to reduce server load, and new stories were reaching Google late. Cutting the cache time to seconds largely fixed it. Use caching, but keep cache times short for pages that need to stay current, especially the sitemap.

9. llms.txt

llms.txt is one of the most talked-about and least understood files of the past two years. Jeremy Howard proposed it on September 3, 2024 (llmstxt.org). It’s a Markdown file placed in the site’s root directory, containing a short summary of the site and links to its important pages. The idea is that an AI assistant helping a user can understand the site quickly. The author writes that he expects the file to be used at the moment an answer is generated rather than for training.

It’s a sensible idea on paper. Here’s what’s known about whether it’s used.

Google’s John Mueller said in June 2025, “FWIW no AI system currently uses llms.txt,” meaning no AI system was using the file, which he said could be seen in server logs (Search Engine Roundtable). In June 2026 he described the file’s status as “purely speculative for now” (Search Engine Journal). Google’s AI guide is even more direct: text files made for AI and similar machine-readable files neither hurt nor help your visibility in Google Search, because Google Search ignores them (Google Search Central).

The signal isn’t entirely one-sided. The same report notes that the documentation for Lighthouse, Chrome’s page auditing tool, includes an llms.txt check. Mueller’s view is that if an AI platform that sends you customers asks for the file, it’s worth making. OpenAI’s and Anthropic’s crawler documentation doesn’t mention llms.txt; both describe control through robots.txt.

On the data side, in an SE Ranking study of about 300,000 domains, 10.13% of sites had an llms.txt, and no relationship was found between having the file and how often a site was cited in AI answers. The same report says that in some logs GPTBot occasionally requested llms.txt, but rarely (Search Engine Journal).

Today llms.txt isn’t a tool that increases your AI visibility. It doesn’t replace robots.txt, a good site structure or content that sits in the initial HTML.

I’m not saying never make one. The file is cheap and harmless. It may help on sites developers use alongside AI tools, like software documentation or API docs. But it’s a low priority. A filling machine manufacturer would get far more from spending that time moving its spec table into the initial HTML.

The best test, again, is your logs. If you have an llms.txt, check for a month who requests it. If no verified AI bot does, you have your answer for your own site.

In the same conversation, Mueller was more positive about another approach: WebMCP, a proposal Google supports that aims to let AI agents use a site’s functions directly. We get to protocols like this for agents in Chapter 15.

10. A monthly bot check

Bot access isn’t something you set once and forget. In the past two years new bots have appeared, CDNs have changed their defaults several times, and platforms have updated bot names and versions. Your site changes too; a new theme, a security plugin or a server migration can start turning bots away without anyone noticing. Once a month, it’s enough to:

• Compare the live robots.txt with last month’s version. If there’s a change you didn’t make, find where it came from.
• Check that your CDN and security settings for AI bots still match the decision you made deliberately.
• Pull request counts, status code breakdowns and the most crawled pages for each bot. If a search bot’s requests suddenly dropped or its error rate rose, find out why.
• If you see a new user-agent you don’t recognize in the logs, find out who owns it and what it does, and update your rules if needed.
• Monitor whether the site is up, continuously. One of our clients’ sites was down for about an hour; since then we monitor uptime for all our clients. A bot that arrives during an outage sees an error instead of a page. For a user bot, that means the person asking at that moment can’t get an answer about you.

We’ll add these checks to the monthly routine we build in Chapter 17.

11. Knowing who’s at the door

AI bots aren’t one group, and giving them all the same answer is a mistake. Training bots relate to your brand’s future memory, search bots to your visibility today, and user bots to the person asking a question right now. For most businesses, a reasonable starting point is to keep search and user bots open and decide on training bots according to the business model.

You find out whether that decision is actually in effect by looking at the logs, not at robots.txt. Checking once a month which bots came, what status codes they got and whether they really are who they claim to be will do more for you than anything else in this chapter.

Checklist

Sources

• OpenAI: Overview of OpenAI crawlers
• Anthropic: Does Anthropic crawl data from the web? (last updated April 7, 2026)
• Perplexity: Perplexity crawlers
• Apple: About Applebot
• Google Search Central: Verify requests from Google crawlers and fetchers (last updated March 20, 2026)
• Google Search Central: How Google interprets the robots.txt specification (last updated October 23, 2026)
• PPC Land: Google-Agent joins the crawler list (March 23, 2026)
• Cloudflare: The crawl-to-click gap (August 29, 2025)
• Cloudflare: Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives (August 4, 2025)
• TechRepublic: Cloudflare accuses Perplexity of stealth crawling (Perplexity’s response)
• llmstxt.org: The /llms.txt file (proposed September 3, 2024)
• Search Engine Roundtable: Google, no AI system currently uses llms.txt (June 18, 2025)
• Search Engine Journal: Google says llms.txt is purely speculative for now (June 2, 2026)
• Search Engine Journal: llms.txt shows no clear effect on AI citations based on 300k domains (November 20, 2025)
• Google Search Central: Reduce the Google crawl rate (last updated October 6, 2026)
• Google Search Central: Optimizing for generative AI search (last updated July 10, 2026)

Previous: Chapter 8, AI Traffic and Business Results · Table of contents · Next: Chapter 10, Readable and Citable Pages