How AI Web Search Crawlers Access Your Website
Learn how AI web search crawlers find and read your pages, how search bots differ from training bots and how to control and verify access.
AI web search does not send one universal crawler to your website. An answer may draw on a previously built search index, a page fetched during a conversation or results supplied by another search provider. Each route leaves different evidence in your logs and may follow different access controls.
For publishers, the distinction matters. Allowing a training crawler does not guarantee search visibility. Blocking one provider’s bot does not necessarily prevent another search service from discovering your pages. And a successful HTTP response does not prove that an assistant extracted your content or cited it.
The useful model is a chain: discovery, access, extraction, retrieval and citation. Understanding those stages helps you decide what to permit, what to protect and where to investigate when your content is missing from an answer.
Three routes from a question to your website
Background crawling builds a reusable index
A crawler can discover URLs through links, sitemaps and previously known pages, then request them before anyone asks a relevant question. The provider processes eligible content into an index or another retrieval system that can be searched later.
This separates the website visit from the answer. A crawler might fetch an article on Monday and use an indexed representation in a response on Thursday. The answer therefore need not produce a fresh request to your server, and a recent edit may not immediately appear in the retrieved material.
User-triggered retrieval requests a page during a task
Some assistants retrieve pages when a user supplies a URL or asks a question that needs additional evidence. That request may come from an identifiable agent, a browser service or other retrieval infrastructure.
For AI web search, this creates an access path distinct from background indexing. A page might be reachable during a conversation without being included in a provider’s search index. Conversely, an indexed page might fail a live fetch because its firewall configuration has changed. User-triggered retrieval also does not necessarily mean a complete browser session with JavaScript execution.
Search partners provide another discovery path
An assistant may receive search results or source material from an external search provider instead of independently crawling every source. Its answer can therefore depend on another company’s discovery, indexing and freshness decisions.
A citation without a corresponding visit from the assistant’s named crawler is not automatically suspicious. The source may have entered through a partner index or an earlier fetch. Our explanation of how an AI search engine processes a query covers the query and retrieval side of this process; the access question is which infrastructure actually requested your page.
Identify AI web search bots separately from training bots
Before editing access rules, identify the purpose of each documented agent. Provider names alone are insufficient because one company can operate separate agents for search, training and user-triggered visits.
OpenAI’s crawler documentation distinguishes these functions, while Perplexity’s bot documentation describes its search and user-request agents. Consult current documentation before deploying rules because agent behavior and verification details can change.
| Agent or control | Documented purpose | Practical implication |
|---|---|---|
| OAI-SearchBot | Crawling for OpenAI search features | Treat search access separately from training permission. |
| GPTBot | Crawling content that may be used for model training | Blocking it is not the same as blocking OAI-SearchBot. |
| ChatGPT-User | Certain user-triggered page visits | A request does not establish background indexing. |
| PerplexityBot | Crawling for Perplexity search | Evaluate its access separately from user-triggered retrieval. |
| Perplexity-User | Fetching associated with user requests | Do not assume it behaves like a scheduled crawler. |
| Googlebot | Crawling for Google Search | Google Search eligibility remains relevant to its AI features. |
| Google-Extended | A robots.txt control token for certain other Google AI uses | It is not a separate HTTP crawler or an opt-out from Google Search AI features. |
Google’s guidance for AI features in Search explains the relationship between ordinary Search eligibility and AI Overviews or AI Mode. Avoid treating every product containing “AI” as one access category.
What happens between a request and usable content
The network and security layer decide whether a fetch succeeds
A request must reach your infrastructure before the provider can read anything. DNS resolution, TLS negotiation, CDN rules, rate limits and origin availability can all affect that journey.
Common failures include a 403 denial, a 429 rate limit, a redirect loop or a browser challenge requiring JavaScript or cookies. Some challenge pages return 200 OK, so a successful status code alone is weak evidence. The response body might contain “verify you are human” rather than your article.
For AI web search, a challenge at the edge can make an otherwise public page practically inaccessible. Inspect CDN or firewall events alongside origin logs: a request blocked before reaching the origin will not appear in the origin’s access log.
Fetching HTML is not the same as rendering a browser page
Retrieval systems differ in their ability to execute JavaScript, wait for asynchronous requests and interact with page controls. Do not assume that every agent sees what a visitor sees after hydration, scrolling or clicking a tab.
Make essential public information available in the initial HTML where practical. That includes the main explanation, descriptive headings and links to related pages. If the initial response contains only an application shell, an agent without suitable rendering support may receive almost no useful text.
This does not mean JavaScript frameworks are inherently inaccessible. Server rendering or prerendering can expose substantial content before client-side execution. The relevant question is what the specific request receives, not which framework generated the site.
Extraction, indexing and citation are separate decisions
After fetching a page, a system may remove navigation, identify the main text, resolve duplicates and associate the content with a canonical URL. Some systems retrieve smaller passages rather than passing an entire page into answer generation.
A page that survives extraction still has no guaranteed place in an answer. Relevance, freshness, source selection and the wording of the question all affect what gets retrieved and cited.
That is why AI web search visibility cannot be reduced to an allowlist. Accessible pages need clear, self-contained explanations that support specific claims. When checking a citation, verify that the linked page actually supports the generated statement rather than merely mentioning the same topic.
Use robots.txt for crawl preferences, not security
For cooperating crawlers, robots.txt communicates which paths they may request. It does not authenticate visitors, prevent URL discovery or protect confidential material.
A narrow example can separate OpenAI search access from training crawling:
User-agent: OAI-SearchBot
Disallow: /account/
Disallow: /admin/
User-agent: GPTBot
Disallow: /
This example leaves other paths available to OAI-SearchBot while disallowing GPTBot crawling. It expresses preferences for those agents only. It does not establish a policy for every provider, remove previously collected material or prevent unauthorized clients from ignoring the file.
Merge rules into your existing configuration rather than replacing it blindly. A bot-specific group can take precedence over wildcard rules, so preserve required path restrictions in the applicable groups. Also ensure robots.txt itself is reachable without a login or browser challenge.
User-triggered agents need particular care. Some providers document that robots.txt rules may not apply to certain user-initiated visits. Check the relevant provider’s policy rather than assuming that every request obeys background-crawler rules.
Controlling AI web search also requires distinguishing crawling from indexing. A supported noindex directive addresses indexing, but a crawler generally needs access to read it. Snippet controls are provider-specific, and neither they nor robots.txt should replace authentication for private content. An llms.txt file is not a universal access requirement or a substitute for these controls.

Verify the visitor and the response separately
A user-agent string is a claim, not proof
Any HTTP client can send a familiar crawler name. If you create firewall exceptions based only on that name, unrelated traffic can impersonate the allowed agent.
Use the provider’s documented verification method, such as published IP ranges where available. If its method involves reverse DNS, perform the corresponding forward verification rather than trusting a hostname alone. Behind a proxy or CDN, use trusted client-IP information from your infrastructure, not arbitrary headers supplied by the requester.
Verification answers who made the request. Response inspection answers what they received. Both are necessary before concluding that an access rule works.
Inspect what your server actually returns
A command-line request can expose redirects, challenge pages and missing HTML. For example:
curl -sS -L -D headers.txt \
-A 'OAI-SearchBot' \
'https://your-domain.example/article' \
-o body.html
This tests how your site responds to that user-agent string from your machine. It does not reproduce the provider’s verified infrastructure, IP reputation or rendering behavior. A successful local test cannot prove that the real crawler is allowed.
For AI web search troubleshooting, compare the test with verified requests in edge and origin logs. Record the URL, timestamp, status, response size and relevant security action. Inspect the returned content when possible, especially when the status suggests success but the body is unexpectedly small.
Diagnose the stage that failed
Treat access, indexing and citation as separate hypotheses. Otherwise, teams often change firewall rules to solve a source-selection problem or rewrite content when requests are actually being blocked.
| Observation | What it establishes | What to investigate next |
|---|---|---|
A verified bot receives 403 |
That request was denied | Firewall, CDN and authorization rules |
A request receives 200 with challenge text |
The article was not successfully delivered | Bot protection and challenge configuration |
| A verified bot receives the article HTML | The fetch succeeded at that time | Extraction, indexing eligibility and canonical handling |
| A page is fetched but not cited | Access alone did not produce a citation | Query relevance, source selection and passage quality |
| A citation appears without a matching bot visit | No matching visit was observed in the available logs | Earlier crawls, partner indexes, caches and log coverage |
Use a small set of representative URLs rather than repeatedly changing sitewide settings. Include a stable reference page, a recently updated article and a JavaScript-heavy page. Their different behavior can reveal whether the problem is global or template-specific.
Our AI SEO crawl audit checklist for publishers provides a structured way to document these checks. Keep the evidence attached to each finding so that a later citation change does not get mistaken for proof that a technical fix worked.
Keep permissions, privacy and AI governance distinct
A crawler access policy should state which public content you want discovered and which routes require authentication. Coordinate with whoever manages the CDN so that a robots.txt change is not silently contradicted by an edge rule.
For AI web search, permission to retrieve a public page is a different question from permission to process customer records, reveal account data or reuse content under contractual terms. Sensitive information belongs behind enforceable access controls, regardless of which bot names appear in robots.txt.
AI-related engineering decisions also need their own classification. If your developers use an assistant to write crawler-gateway code, that fact alone does not determine the software’s regulatory status. This explanation of EU AI Act classification for AI-assisted software distinguishes the system, intended use and organizational role from the mere use of a coding assistant. Keep that assessment separate from your crawler permission record.
Frequently asked questions
Does blocking GPTBot stop my site from appearing in ChatGPT search? Not by itself. OpenAI documents GPTBot and OAI-SearchBot as separate agents with different purposes. Review search access independently, and remember that other retrieval paths may exist.
Do AI crawlers execute JavaScript? Capabilities vary across providers and retrieval modes. Test the response delivered to the relevant agent rather than assuming it behaves like a fully interactive browser. Essential public text in initial HTML reduces reliance on rendering support.
Does a crawler visit guarantee a citation? No. A visit establishes an access event, not successful indexing, retrieval or source selection. Even a fully accessible page may not be relevant to the question being answered.
Can AI web search use a page without visiting it during the conversation? Yes. An answer may use material retrieved from an existing index, a cache or a search partner. Your logs therefore cannot provide a complete account of every downstream use.
Can I protect private pages with robots.txt? No. Use authentication and authorization. Robots.txt is public, can disclose path names and relies on cooperating clients to follow its instructions.
Start with one URL and an explicit access policy
Choose one important public page, identify the agents you want to allow and confirm what verified requests receive. Then investigate extraction and search eligibility before treating missing citations as an access failure.
Once access is understood, measure the business outcome separately. Evaluating whether AI search sends qualified traffic helps distinguish technical availability from visits that actually matter to your organization.
Read more

How OpenAI Search Handles Paywalls and Subscriber Content
A paywalled article can appear in an AI answer without its full text being available. Learn how citations, publisher agreements and crawler controls differ, then protect subscriber content without hiding every useful preview.

How to Manage Your Privacy in ChatGPT Search
Turning off model training does not make web searches anonymous or erase saved information. Learn which settings control memory, location and chat retention, plus what to check before opening a result or searching with work data.

How ChatGPT and Google Handle Fresh Information
An answer can sound current while relying on old evidence. Learn how each platform finds updates, how to check time-sensitive claims and what publishers can do to make genuine changes easier to discover.