Answer engine optimization works by aligning content with each stage of the pipeline AI answer engines run: crawlers discover and fetch pages [4][6], a search index or training corpus ingests them [2], a retrieval step pulls candidate passages when someone asks a question [1][9], the model grounds its generated answer in those passages [8], and a citation layer credits the small set of sources that shaped the response [5]. Platforms document the access and eligibility rules for the early stages: a page must be crawlable by the right bots, and on Google it must be indexed and snippet-eligible before it can appear as a supporting link [1][4]. The selection logic of the later stages is not published, so practitioners work from independent evidence, such as a controlled benchmark in which adding quotations, statistics, and cited sources raised visibility in generated answers by as much as 40% [10]. Effective AEO treats each stage as a filter and fixes the earliest failing stage first, because a page that never gets crawled or indexed cannot be retrieved, grounded on, or cited [1][2].
\n","summary":"A stage-by-stage walkthrough of the pipeline behind AI answers: which crawlers fetch your pages, how content enters a search index or a training corpus, how query fan-out retrieval selects candidate passages, how grounding ties generated sentences to specific sources, and what evidence exists about citation selection. Includes a practice-to-stage map and flags, for every claim, whether it rests on platform documentation or on practitioner inference.","publishedAt":"2026-08-09T20:58:23.191","verifiedAt":"2026-08-09T00:00:00","editorialStatus":"APPROVED","lastReviewedAt":"2026-08-09T00:00:00","nextReviewDueAt":"2026-11-09T00:00:00","templateVersion":"v2","aliases":["How does AEO work mechanically","How AI answer engines pick sources","How do AI answer engines choose citations","How AI search engines crawl and cite content","How does RAG work in AI search","What is query fan-out in AI search","How ChatGPT selects sources to cite","How AI Overviews choose which links to show","How do LLMs decide what to cite","Answer engine pipeline explained","How generative engines retrieve and cite content","AEO mechanics stage by stage"],"confidenceScore":86,"confidenceLabel":"High","canonicalUrl":null},"contributor":{"id":"ec39deab-44fe-48d8-9029-fefe993ab85a","slug":"answer-stack","displayName":"AnswerStack","websiteUrl":null},"contributorOrganizationProfile":{"entityId":"ec39deab-44fe-48d8-9029-fefe993ab85a","legalName":null,"description":null,"websiteUrl":null,"imageUrl":null,"slogan":null,"subtitle":null,"facts":[],"coiNote":null,"foundingDate":null,"numberOfEmployeesText":null,"contactPoint":null,"address":null,"headquartersText":null,"organizationType":null},"contributorPerson":{"slug":"answerstack-editorial-team","displayName":"AnswerStack Editorial Team"},"sections":[{"id":"47085fb0-6781-4594-b223-1d1bf8a8d050","sectionKey":"pipeline_overview","sectionType":"markdown_section","heading":"What happens between publishing a page and seeing it cited?","introMarkdown":"Every major answer engine runs the same basic pipeline, and AEO is the work of clearing each stage in order. First a crawler has to discover and fetch the page [2][4][6]. The content is then ingested, either into a search index that updates continuously or into a training corpus that shapes what a model knows from memory [2][4]. When someone asks a question, a retrieval step selects candidate documents, often by running several related searches instead of one [1][8]. The model generates an answer grounded in those retrieved passages [8], and a final layer determines which of the sources get named and linked as citations [5][11]. A page that fails an early stage never reaches a later one, which is why diagnosing AEO problems in pipeline order beats guessing at content changes.\n\nTwo supply routes feed these systems, and they behave differently. The first route is parametric: content collected by training crawlers ends up encoded in model weights, which is why a chatbot can describe a brand without searching at all [4][7]. The research that shaped current systems calls this parametric memory and pairs it with a non-parametric route, a live document index the model queries at answer time, an architecture known as retrieval-augmented generation, or RAG [9]. Citations almost always come from the second route, because an engine can point at a document it just retrieved but cannot point at the diffuse training data behind a memorized fact [5][8].\n\nOne more distinction runs through this whole answer. Platforms publish genuine documentation for the early stages: crawler names and purposes, robots.txt behavior, and eligibility rules [1][4][6][7]. They publish almost nothing about how the later stages score and select sources, so claims about citation selection rest on independent studies and practitioner testing rather than official specifications [10][11]. Each section below states which kind of evidence it stands on.","introHtml":"Every major answer engine runs the same basic pipeline, and AEO is the work of clearing each stage in order. First a crawler has to discover and fetch the page [2][4][6]. The content is then ingested, either into a search index that updates continuously or into a training corpus that shapes what a model knows from memory [2][4]. When someone asks a question, a retrieval step selects candidate documents, often by running several related searches instead of one [1][8]. The model generates an answer grounded in those retrieved passages [8], and a final layer determines which of the sources get named and linked as citations [5][11]. A page that fails an early stage never reaches a later one, which is why diagnosing AEO problems in pipeline order beats guessing at content changes.
\nTwo supply routes feed these systems, and they behave differently. The first route is parametric: content collected by training crawlers ends up encoded in model weights, which is why a chatbot can describe a brand without searching at all [4][7]. The research that shaped current systems calls this parametric memory and pairs it with a non-parametric route, a live document index the model queries at answer time, an architecture known as retrieval-augmented generation, or RAG [9]. Citations almost always come from the second route, because an engine can point at a document it just retrieved but cannot point at the diffuse training data behind a memorized fact [5][8].
\nOne more distinction runs through this whole answer. Platforms publish genuine documentation for the early stages: crawler names and purposes, robots.txt behavior, and eligibility rules [1][4][6][7]. They publish almost nothing about how the later stages score and select sources, so claims about citation selection rest on independent studies and practitioner testing rather than official specifications [10][11]. Each section below states which kind of evidence it stands on.
\n","outroMarkdown":null,"outroHtml":null,"contentJson":{},"configJson":{},"noteMarkdown":null,"noteHtml":null,"sortOrder":0},{"id":"7f3f31e5-b512-40cf-be9a-caf0d56556fe","sectionKey":"pipeline_stage_table","sectionType":"table_section","heading":"What are the stages of the answer engine pipeline?","introMarkdown":"Five stages sit between a published page and a citation in an AI answer. The table summarizes them, and each stage gets its own section below.","introHtml":"Five stages sit between a published page and a citation in an AI answer. The table summarizes them, and each stage gets its own section below.
\n","outroMarkdown":"Each stage filters out some share of the pages that entered the one before it. The practical consequence is an ordering rule: confirm crawl access before worrying about indexing, confirm indexing before worrying about retrieval, and only then invest in the content-level work that influences grounding and citation.","outroHtml":"Each stage filters out some share of the pages that entered the one before it. The practical consequence is an ordering rule: confirm crawl access before worrying about indexing, confirm indexing before worrying about retrieval, and only then invest in the content-level work that influences grounding and citation.
\n","contentJson":{"rows":[{"cells":["Discovery and crawling","Bots fetch pages: search crawlers, training crawlers, and user-triggered fetchers","Google, OpenAI, Perplexity, and Anthropic all publish crawler docs [2][4][6][7]","Allow the right bots in robots.txt, deliberately"]},{"cells":["Indexing and ingestion","Content enters a live search index or a training corpus; indexing is not guaranteed","Google documents indexing in detail; the AI labs document training crawls [2][3][4]","Serve indexable, renderable pages that return HTTP 200"]},{"cells":["Retrieval","The engine runs one or many searches (query fan-out) and pulls candidate passages","Google and OpenAI document the mechanism [1][5][8]","Answer specific sub-questions, not just head terms"]},{"cells":["Grounding and generation","The model composes an answer based on retrieved content to reduce hallucination","Google and OpenAI developer docs [5][8]","Write self-contained passages that express one claim completely"]},{"cells":["Citation selection","A subset of retrieved sources gets named and linked in the answer","Formats and eligibility are documented; scoring is not [1][5][6]","Evidence density: statistics, quotations, and cited sources [10]"]}],"columns":["Stage","What happens","Who documents it","The AEO lever"]},"configJson":{},"noteMarkdown":null,"noteHtml":null,"sortOrder":1},{"id":"60a6c21e-e2a8-4b3f-a8d9-ce58e6db9a0f","sectionKey":"discovery_crawling","sectionType":"markdown_section","heading":"How do answer engines discover and crawl content?","introMarkdown":"Three classes of bots touch a website, and each has a different job and a different control. Training crawlers such as OpenAI's GPTBot and Anthropic's ClaudeBot collect content that may be used to train foundation models [4][7]. Search-index crawlers exist to power live answers and citations: Googlebot feeds the index behind AI Overviews and AI Mode [1][2], OAI-SearchBot is used to surface websites in ChatGPT's search features [4], PerplexityBot exists to surface and link websites in Perplexity's results and is not used to gather training data [6], and Anthropic's Claude-SearchBot analyzes content to improve the relevance and accuracy of Claude's search responses [7]. The third class, user-triggered fetchers such as ChatGPT-User and Perplexity-User, retrieves a page on demand when an individual user asks about it, and because a person initiated the request these fetchers may not honor robots.txt: Perplexity states plainly that its user-triggered fetcher generally ignores robots.txt rules [4][6].\n\nThe classes are controlled independently, and that separation drives real decisions. OpenAI documents that a site can block GPTBot to keep content out of model training while still allowing OAI-SearchBot so its pages appear in ChatGPT search answers, and notes that robots.txt changes take roughly 24 hours to register with its systems [4]. Sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers at all, though they can still appear as navigational links [4]. The concrete AEO task at this stage is an access audit: read the robots.txt file, identify which of the three bot classes each rule affects, and confirm that every block reflects a current decision rather than a blanket AI ban added in an earlier policy cycle.","introHtml":"Three classes of bots touch a website, and each has a different job and a different control. Training crawlers such as OpenAI's GPTBot and Anthropic's ClaudeBot collect content that may be used to train foundation models [4][7]. Search-index crawlers exist to power live answers and citations: Googlebot feeds the index behind AI Overviews and AI Mode [1][2], OAI-SearchBot is used to surface websites in ChatGPT's search features [4], PerplexityBot exists to surface and link websites in Perplexity's results and is not used to gather training data [6], and Anthropic's Claude-SearchBot analyzes content to improve the relevance and accuracy of Claude's search responses [7]. The third class, user-triggered fetchers such as ChatGPT-User and Perplexity-User, retrieves a page on demand when an individual user asks about it, and because a person initiated the request these fetchers may not honor robots.txt: Perplexity states plainly that its user-triggered fetcher generally ignores robots.txt rules [4][6].
\nThe classes are controlled independently, and that separation drives real decisions. OpenAI documents that a site can block GPTBot to keep content out of model training while still allowing OAI-SearchBot so its pages appear in ChatGPT search answers, and notes that robots.txt changes take roughly 24 hours to register with its systems [4]. Sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers at all, though they can still appear as navigational links [4]. The concrete AEO task at this stage is an access audit: read the robots.txt file, identify which of the three bot classes each rule affects, and confirm that every block reflects a current decision rather than a blanket AI ban added in an earlier policy cycle.
\n","outroMarkdown":null,"outroHtml":null,"contentJson":{},"configJson":{},"noteMarkdown":null,"noteHtml":null,"sortOrder":2},{"id":"dab4216d-7861-484d-8c6d-019b3dbf7845","sectionKey":"indexing_ingestion","sectionType":"markdown_section","heading":"What happens when content is ingested?","introMarkdown":"Crawled content moves into storage next, and the two ingestion routes run on different clocks. Google documents its route in detail: Googlebot renders pages and executes JavaScript, the system clusters duplicates and selects a canonical version, and the result may be stored in an index hosted across thousands of machines, with the explicit caveat that indexing is not guaranteed and not every processed page makes it in [2]. Google's stated technical bar for its AI features is identical to classic search: the page must be reachable by Googlebot, return an HTTP 200 status code, and contain indexable content [3]. Eligibility then follows a single documented rule: to appear as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet [1].\n\nThe training route is slower and less visible. Content collected by GPTBot or ClaudeBot may inform a future model's weights, and knowledge stored that way stays fixed until the next training run [4][7][9]. This split explains a pattern that confuses many teams: a chatbot can describe a company fluently from memory while never citing its site, or cite the site in search mode while its memorized description remains outdated. AEO work at this stage is largely traditional technical SEO, which is not a coincidence, since Google's AI features draw on the same index as classic search and the other engines built comparable search indexes of their own [1][3][6]. Fixing broken status codes, blocked resources, unrendered JavaScript content, and stray noindex directives clears the ingestion stage for every engine that relies on an index.","introHtml":"Crawled content moves into storage next, and the two ingestion routes run on different clocks. Google documents its route in detail: Googlebot renders pages and executes JavaScript, the system clusters duplicates and selects a canonical version, and the result may be stored in an index hosted across thousands of machines, with the explicit caveat that indexing is not guaranteed and not every processed page makes it in [2]. Google's stated technical bar for its AI features is identical to classic search: the page must be reachable by Googlebot, return an HTTP 200 status code, and contain indexable content [3]. Eligibility then follows a single documented rule: to appear as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet [1].
\nThe training route is slower and less visible. Content collected by GPTBot or ClaudeBot may inform a future model's weights, and knowledge stored that way stays fixed until the next training run [4][7][9]. This split explains a pattern that confuses many teams: a chatbot can describe a company fluently from memory while never citing its site, or cite the site in search mode while its memorized description remains outdated. AEO work at this stage is largely traditional technical SEO, which is not a coincidence, since Google's AI features draw on the same index as classic search and the other engines built comparable search indexes of their own [1][3][6]. Fixing broken status codes, blocked resources, unrendered JavaScript content, and stray noindex directives clears the ingestion stage for every engine that relies on an index.
\n","outroMarkdown":null,"outroHtml":null,"contentJson":{},"configJson":{},"noteMarkdown":null,"noteHtml":null,"sortOrder":3},{"id":"f0619733-8295-4c5d-bbc1-78221686a338","sectionKey":"retrieval","sectionType":"markdown_section","heading":"How does retrieval work when someone asks a question?","introMarkdown":"Retrieval turns a stored page into a candidate for an answer, and it is better documented than most practitioners assume. Google states that AI Mode and AI Overviews may use a query fan-out technique, issuing multiple related searches across subtopics and data sources to develop a single response [1]. Its Gemini developer documentation describes the same loop at the model level: the model analyzes the prompt, determines whether a search would improve the answer, and if so automatically generates one or multiple search queries and executes them [8]. OpenAI documents three retrieval depths for its search tooling: a basic mode that answers from top results, an agentic mode in which the model searches inside its chain of thought and decides whether to keep searching, and deep research, which often draws on hundreds of sources for one task [5].\n\nFan-out changes what ranking means in practice. An Ahrefs analysis of four million AI Overview citations, updated in March 2026, found that only 37.9% of cited URLs appeared within the first ten result blocks for the original query, with the remainder split almost evenly between positions 11 to 100 and beyond the top 100 [11]. Ahrefs attributes the spread to fan-out: pages that rank for the related sub-queries get cited even when they never rank for the visible head query [11]. The AEO mapping follows directly. A page that answers a specific sub-question, such as a pricing detail or an implementation step, can be retrieved by one of the fanned-out searches without ever competing for the broad term, which makes question-level topic coverage a retrieval tactic rather than a stylistic preference.","introHtml":"Retrieval turns a stored page into a candidate for an answer, and it is better documented than most practitioners assume. Google states that AI Mode and AI Overviews may use a query fan-out technique, issuing multiple related searches across subtopics and data sources to develop a single response [1]. Its Gemini developer documentation describes the same loop at the model level: the model analyzes the prompt, determines whether a search would improve the answer, and if so automatically generates one or multiple search queries and executes them [8]. OpenAI documents three retrieval depths for its search tooling: a basic mode that answers from top results, an agentic mode in which the model searches inside its chain of thought and decides whether to keep searching, and deep research, which often draws on hundreds of sources for one task [5].
\nFan-out changes what ranking means in practice. An Ahrefs analysis of four million AI Overview citations, updated in March 2026, found that only 37.9% of cited URLs appeared within the first ten result blocks for the original query, with the remainder split almost evenly between positions 11 to 100 and beyond the top 100 [11]. Ahrefs attributes the spread to fan-out: pages that rank for the related sub-queries get cited even when they never rank for the visible head query [11]. The AEO mapping follows directly. A page that answers a specific sub-question, such as a pricing detail or an implementation step, can be retrieved by one of the fanned-out searches without ever competing for the broad term, which makes question-level topic coverage a retrieval tactic rather than a stylistic preference.
\n","outroMarkdown":null,"outroHtml":null,"contentJson":{},"configJson":{},"noteMarkdown":null,"noteHtml":null,"sortOrder":4},{"id":"d90c3953-a954-4257-9d7c-f30ee0159980","sectionKey":"grounding_generation","sectionType":"markdown_section","heading":"How do grounding and answer generation work?","introMarkdown":"Grounding means the model bases its generated answer on retrieved documents instead of memory alone. Google's developer documentation describes grounding with Google Search as connecting the model to real-time web content so it can provide more accurate answers, cite verifiable sources beyond its knowledge cutoff, and reduce hallucinations by basing responses on real-world information [8].\n\nThe mechanics operate at the level of individual spans of text. Gemini's API returns citation annotations that carry a start index and end index identifying exactly which stretch of generated text each source supports [8]. OpenAI's url_citation annotations work the same way, carrying the URL, title, and location of the cited source, together with a policy requirement that inline citations be kept clearly visible and clickable when web results are shown to users [5]. In other words, the citation is attached to a specific claim in the output, not to the answer as a whole.\n\nSpan-level attribution is the likely reason the standard AEO advice to write self-contained passages exists. If an engine ties a source to one generated sentence, the passage it retrieved needs to express that claim completely, without leaning on the rest of the page for meaning. That reasoning is practitioner inference, not documented platform behavior, and it deserves that label. It is, however, consistent with the one controlled benchmark in the field, where rewriting pages so claims are stated directly with supporting evidence nearby measurably increased their presence in generated answers [10].","introHtml":"Grounding means the model bases its generated answer on retrieved documents instead of memory alone. Google's developer documentation describes grounding with Google Search as connecting the model to real-time web content so it can provide more accurate answers, cite verifiable sources beyond its knowledge cutoff, and reduce hallucinations by basing responses on real-world information [8].
\nThe mechanics operate at the level of individual spans of text. Gemini's API returns citation annotations that carry a start index and end index identifying exactly which stretch of generated text each source supports [8]. OpenAI's url_citation annotations work the same way, carrying the URL, title, and location of the cited source, together with a policy requirement that inline citations be kept clearly visible and clickable when web results are shown to users [5]. In other words, the citation is attached to a specific claim in the output, not to the answer as a whole.
\nSpan-level attribution is the likely reason the standard AEO advice to write self-contained passages exists. If an engine ties a source to one generated sentence, the passage it retrieved needs to express that claim completely, without leaning on the rest of the page for meaning. That reasoning is practitioner inference, not documented platform behavior, and it deserves that label. It is, however, consistent with the one controlled benchmark in the field, where rewriting pages so claims are stated directly with supporting evidence nearby measurably increased their presence in generated answers [10].
\n","outroMarkdown":null,"outroHtml":null,"contentJson":{},"configJson":{},"noteMarkdown":null,"noteHtml":null,"sortOrder":5},{"id":"cbc751f0-dc16-466d-bbaa-3cc0172fd52b","sectionKey":"citation_selection","sectionType":"markdown_section","heading":"How do answer engines choose which sources to cite?","introMarkdown":"Platforms document the eligibility gate and the citation format, not the scoring. Google's documented rule is that a supporting link must be indexed and snippet-eligible, and that the standard preview controls, including nosnippet, data-nosnippet, max-snippet, and noindex, limit what its AI features may display [1][3]. OpenAI documents that model responses include inline citations for URLs found in web search results by default [5]. Perplexity's documentation establishes that its crawler exists specifically to surface and link websites in results [6]. None of the three publishes a formula for why one eligible page gets the citation over another.\n\nIndependent evidence fills part of that gap. The GEO benchmark, first released in November 2023 and presented at KDD 2024, tested nine content strategies across thousands of queries and measured which raised a source's share of the generated answer [10]. Adding quotations improved position-adjusted visibility by about 41%, and adding statistics or citing sources each improved it by roughly 30%, while keyword stuffing, a classic search-era tactic, offered little to no improvement and sometimes performed below baseline [10]. The authors also found effectiveness varied by domain, which argues against copying any single tactic list blindly [10].\n\nObservational studies add a second layer, with weaker but broader evidence. The Ahrefs citation data shows the mix of cited sources shifting substantially between mid-2025 and early 2026, and shows platforms drawing heavily on pages outside the top organic results [11]. Read together, the documented rules and the independent measurements support a working model: eligibility is binary and verifiable, while selection favors passages that carry checkable evidence, and the exact weighting changes as the products change [1][10][11].","introHtml":"Platforms document the eligibility gate and the citation format, not the scoring. Google's documented rule is that a supporting link must be indexed and snippet-eligible, and that the standard preview controls, including nosnippet, data-nosnippet, max-snippet, and noindex, limit what its AI features may display [1][3]. OpenAI documents that model responses include inline citations for URLs found in web search results by default [5]. Perplexity's documentation establishes that its crawler exists specifically to surface and link websites in results [6]. None of the three publishes a formula for why one eligible page gets the citation over another.
\nIndependent evidence fills part of that gap. The GEO benchmark, first released in November 2023 and presented at KDD 2024, tested nine content strategies across thousands of queries and measured which raised a source's share of the generated answer [10]. Adding quotations improved position-adjusted visibility by about 41%, and adding statistics or citing sources each improved it by roughly 30%, while keyword stuffing, a classic search-era tactic, offered little to no improvement and sometimes performed below baseline [10]. The authors also found effectiveness varied by domain, which argues against copying any single tactic list blindly [10].
\nObservational studies add a second layer, with weaker but broader evidence. The Ahrefs citation data shows the mix of cited sources shifting substantially between mid-2025 and early 2026, and shows platforms drawing heavily on pages outside the top organic results [11]. Read together, the documented rules and the independent measurements support a working model: eligibility is binary and verifiable, while selection favors passages that carry checkable evidence, and the exact weighting changes as the products change [1][10][11].
\n","outroMarkdown":null,"outroHtml":null,"contentJson":{},"configJson":{},"noteMarkdown":null,"noteHtml":null,"sortOrder":6},{"id":"53ab66ca-3afe-437b-b8fc-b228c9824bac","sectionKey":"practice_stage_map","sectionType":"table_section","heading":"Which AEO practices map to which pipeline stage?","introMarkdown":"Common AEO practices each act on one or two specific stages, and their evidence quality varies widely. The evidence column is the part most vendor content omits.","introHtml":"Common AEO practices each act on one or two specific stages, and their evidence quality varies widely. The evidence column is the part most vendor content omits.
\n","outroMarkdown":"A useful discipline when evaluating any new AEO recommendation is to ask which stage it is supposed to influence and what class of evidence supports it. If a recommendation cannot answer either question, treat it as general content advice rather than a tested AEO tactic.","outroHtml":"A useful discipline when evaluating any new AEO recommendation is to ask which stage it is supposed to influence and what class of evidence supports it. If a recommendation cannot answer either question, treat it as general content advice rather than a tested AEO tactic.
\n","contentJson":{"rows":[{"cells":["Per-bot robots.txt audit","Discovery and crawling","Documented by every major platform [4][6][7]"]},{"cells":["Indexable, renderable pages with clean status codes","Indexing and ingestion","Documented by Google [2][3]"]},{"cells":["Keeping pages snippet-eligible","Citation eligibility","Documented by Google [1]"]},{"cells":["Covering specific sub-questions in depth","Retrieval via query fan-out","Mechanism documented; the tactic is inferred from citation data [1][11]"]},{"cells":["Self-contained, quotable passages","Grounding and generation","Practitioner inference, consistent with benchmark results [5][8][10]"]},{"cells":["Adding statistics, quotations, and cited sources","Citation selection","Controlled benchmark evidence, up to 41% visibility gain [10]"]},{"cells":["Structured data markup","Indexing and rich results","Documented as useful for machine readability, and as optional for AI features [1][3]"]},{"cells":["Creating an llms.txt file","No documented stage","Proposal only; no major engine documents reading it [1][12]"]}],"columns":["AEO practice","Pipeline stage it affects","Evidence status"]},"configJson":{},"noteMarkdown":null,"noteHtml":null,"sortOrder":7},{"id":"a15cce69-6850-41f4-b5f2-4fecf0737461","sectionKey":"documented_vs_inferred","sectionType":"markdown_section","heading":"What is documented and what is practitioner inference?","introMarkdown":"The documented layer is larger than many practitioners realize. Platforms publish their crawler names and purposes, the robots.txt controls for each, the roughly 24-hour propagation delay on OpenAI's side, Google's indexed-and-snippet-eligible rule, the existence of query fan-out, the span-level citation formats, and the display requirements for citations [1][4][5][6][8]. Google goes further and rules things out: its documentation states that you do not need to create new machine-readable files, AI text files, or markup to appear in its AI features, and that no special schema.org structured data is required [1].\n\nThe inferred layer covers everything about weighting. No platform discloses how freshness, authority, entity recognition, or content structure are scored during retrieval and selection, so any confident claim about those factors comes from outside measurement. The honest evidence base consists of one peer-reviewed controlled benchmark [10] and a set of large observational studies from SEO tool vendors [11], which show correlations rather than causes and can shift within months.\n\nTwo cases show why the distinction pays. Structured data is documented as a machine-readable way to describe content that makes pages eligible for rich results, with a guideline that markup must match visible content [3], yet it is explicitly not required for Google's AI features [1]; treating it as disambiguation infrastructure fits the documentation, while treating it as an AI citation lever goes beyond it. The llms.txt file, proposed by Jeremy Howard on September 3, 2024 as a standard way to give language models an inference-time guide to a website, remains an open community proposal [12]; no major answer engine documents consuming it, and Google's guidance on AI text files points the other way [1]. Neither practice is harmful, but only one has documented standing.","introHtml":"The documented layer is larger than many practitioners realize. Platforms publish their crawler names and purposes, the robots.txt controls for each, the roughly 24-hour propagation delay on OpenAI's side, Google's indexed-and-snippet-eligible rule, the existence of query fan-out, the span-level citation formats, and the display requirements for citations [1][4][5][6][8]. Google goes further and rules things out: its documentation states that you do not need to create new machine-readable files, AI text files, or markup to appear in its AI features, and that no special schema.org structured data is required [1].
\nThe inferred layer covers everything about weighting. No platform discloses how freshness, authority, entity recognition, or content structure are scored during retrieval and selection, so any confident claim about those factors comes from outside measurement. The honest evidence base consists of one peer-reviewed controlled benchmark [10] and a set of large observational studies from SEO tool vendors [11], which show correlations rather than causes and can shift within months.
\nTwo cases show why the distinction pays. Structured data is documented as a machine-readable way to describe content that makes pages eligible for rich results, with a guideline that markup must match visible content [3], yet it is explicitly not required for Google's AI features [1]; treating it as disambiguation infrastructure fits the documentation, while treating it as an AI citation lever goes beyond it. The llms.txt file, proposed by Jeremy Howard on September 3, 2024 as a standard way to give language models an inference-time guide to a website, remains an open community proposal [12]; no major answer engine documents consuming it, and Google's guidance on AI text files points the other way [1]. Neither practice is harmful, but only one has documented standing.
\n","outroMarkdown":null,"outroHtml":null,"contentJson":{},"configJson":{},"noteMarkdown":null,"noteHtml":null,"sortOrder":8},{"id":"c3e6e3d7-b632-4167-a5e4-59aac347ba66","sectionKey":"trade_offs","sectionType":"markdown_section","heading":"Trade-offs and what to watch","introMarkdown":"### Selection behavior drifts quickly\n\nAhrefs measured roughly 76% of AI Overview citations coming from top-10 results in July 2025 and roughly 38% by March 2026, a change it attributes to Google expanding fan-out retrieval [11]. Any tactic calibrated to a snapshot of citation behavior decays as the products change, which favors investing in the stable layers: access, indexability, and evidence quality.\n\n### Verifiability is uneven across stages\n\nYou can confirm crawl access from server logs and index status from Search Console, but no tool confirms whether a page entered a retrieval candidate set. The later stages are observable only by sampling prompts repeatedly, and because the model itself decides whether and how much to search, identical questions can ground on different sources in different sessions [5][8].\n\n### The parametric route cannot be edited quickly\n\nA description of your company memorized during training persists until a model is retrained, regardless of what your site says today [4][7]. Robots.txt changes govern future crawls, not existing weights, so correcting a stale memorized fact is a matter of publishing consistent, well-distributed information and waiting out a training cycle.\n\n### Extraction-friendly writing has a failure mode\n\nPages over-fitted to extraction, with every paragraph compressed into a quotable unit, can read as mechanical to human visitors. The benchmark gains came from adding verifiable evidence, not from formatting tricks, and the one tactic closest to mechanical optimization, keyword stuffing, failed outright [10].","introHtml":"Ahrefs measured roughly 76% of AI Overview citations coming from top-10 results in July 2025 and roughly 38% by March 2026, a change it attributes to Google expanding fan-out retrieval [11]. Any tactic calibrated to a snapshot of citation behavior decays as the products change, which favors investing in the stable layers: access, indexability, and evidence quality.
\nYou can confirm crawl access from server logs and index status from Search Console, but no tool confirms whether a page entered a retrieval candidate set. The later stages are observable only by sampling prompts repeatedly, and because the model itself decides whether and how much to search, identical questions can ground on different sources in different sessions [5][8].
\nA description of your company memorized during training persists until a model is retrained, regardless of what your site says today [4][7]. Robots.txt changes govern future crawls, not existing weights, so correcting a stale memorized fact is a matter of publishing consistent, well-distributed information and waiting out a training cycle.
\nPages over-fitted to extraction, with every paragraph compressed into a quotable unit, can read as mechanical to human visitors. The benchmark gains came from adding verifiable evidence, not from formatting tricks, and the one tactic closest to mechanical optimization, keyword stuffing, failed outright [10].
\n","outroMarkdown":null,"outroHtml":null,"contentJson":{},"configJson":{},"noteMarkdown":null,"noteHtml":null,"sortOrder":9},{"id":"377aa78f-298e-4abf-8d86-6010dc5c0cbf","sectionKey":"what_it_is_not","sectionType":"markdown_section","heading":"What this pipeline is not","introMarkdown":"### It is not a single ranked list\n\nRetrieval assembles candidates from multiple fanned-out searches, and a majority of Google AI citations in recent measurement came from pages outside the top ten for the original query [1][11]. Reading AI visibility off an ordinary rank tracker misses most of the mechanism.\n\n### It is not the same system as model training\n\nSearch surfacing and training ingestion run on separate crawlers with separate controls at OpenAI, Perplexity, and Anthropic [4][6][7]. Blocking training bots does not remove a site from AI search answers, and allowing search bots does not opt a site into training.\n\n### It is not a submission process\n\nNo answer engine offers registration, and no documented file or markup buys entry: Google states that no AI text files, special files, or schema types are required for its AI features [1], and the llms.txt proposal remains unadopted by major engines [12]. Eligibility is earned through ordinary crawlability and indexing [1][3].\n\n### It is not deterministic\n\nThe model chooses whether to search, what queries to issue, and which retrieved passages to ground on, so the same question can produce different sources across sessions and model versions [5][8]. Single spot checks prove little in either direction; only repeated sampling shows whether visibility is real.","introHtml":"Retrieval assembles candidates from multiple fanned-out searches, and a majority of Google AI citations in recent measurement came from pages outside the top ten for the original query [1][11]. Reading AI visibility off an ordinary rank tracker misses most of the mechanism.
\nSearch surfacing and training ingestion run on separate crawlers with separate controls at OpenAI, Perplexity, and Anthropic [4][6][7]. Blocking training bots does not remove a site from AI search answers, and allowing search bots does not opt a site into training.
\nNo answer engine offers registration, and no documented file or markup buys entry: Google states that no AI text files, special files, or schema types are required for its AI features [1], and the llms.txt proposal remains unadopted by major engines [12]. Eligibility is earned through ordinary crawlability and indexing [1][3].
\nThe model chooses whether to search, what queries to issue, and which retrieved passages to ground on, so the same question can produce different sources across sessions and model versions [5][8]. Single spot checks prove little in either direction; only repeated sampling shows whether visibility is real.
\n","outroMarkdown":null,"outroHtml":null,"contentJson":{},"configJson":{},"noteMarkdown":null,"noteHtml":null,"sortOrder":10},{"id":"9abfc996-fd22-40df-a7a4-ee20815b0dc5","sectionKey":"contributor_perspective","sectionType":"markdown_section","heading":"How this answer was researched","introMarkdown":"This entry was researched and written by the AnswerStack Editorial Team as independent reference material, with no client, advertiser, or vendor relationship to any platform, tool, or agency named. Every cited URL was fetched and confirmed live on August 9, 2026. Platform documentation from Google, OpenAI, Perplexity, and Anthropic was treated as the standard for documented behavior, and claims that go beyond what those documents say are labeled as inference or as benchmark evidence in the text. Because crawler policies, retrieval techniques, and citation behavior changed repeatedly between 2024 and 2026, the figures here should be read as measurements from the dates cited rather than fixed properties of these systems, and this page carries a scheduled review date of November 9, 2026. Practitioners who have run controlled or well-sampled AEO experiments are invited to contribute corrections and additions through AnswerStack's contributor process.","introHtml":"This entry was researched and written by the AnswerStack Editorial Team as independent reference material, with no client, advertiser, or vendor relationship to any platform, tool, or agency named. Every cited URL was fetched and confirmed live on August 9, 2026. Platform documentation from Google, OpenAI, Perplexity, and Anthropic was treated as the standard for documented behavior, and claims that go beyond what those documents say are labeled as inference or as benchmark evidence in the text. Because crawler policies, retrieval techniques, and citation behavior changed repeatedly between 2024 and 2026, the figures here should be read as measurements from the dates cited rather than fixed properties of these systems, and this page carries a scheduled review date of November 9, 2026. Practitioners who have run controlled or well-sampled AEO experiments are invited to contribute corrections and additions through AnswerStack's contributor process.
\n","outroMarkdown":null,"outroHtml":null,"contentJson":{},"configJson":{},"noteMarkdown":"This answer was written and reviewed by the AnswerStack Editorial Team, which has no commercial stake in the products, companies, or methods discussed. Every claim is cited inline and verified on the dates shown.","noteHtml":"This answer was written and reviewed by the AnswerStack Editorial Team, which has no commercial stake in the products, companies, or methods discussed. Every claim is cited inline and verified on the dates shown.
\n","sortOrder":11}],"citations":[{"title":"AI features and your website","url":"https://developers.google.com/search/docs/appearance/ai-features","excerpt":"To be eligible to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet.","quoteText":null,"sourceRole":"PRIMARY","verifiedAt":"2026-08-09T00:00:00","supportsText":"Indexed and snippet-eligible requirement for AI Overviews and AI Mode supporting links; query fan-out technique; no special files, AI text files, or markup needed; preview controls; Google-Extended","domain":"developers.google.com","publisherName":"Google Search Central"},{"title":"In-depth guide to how Google Search works","url":"https://developers.google.com/search/docs/fundamentals/how-search-works","excerpt":"Indexing isn't guaranteed; not every page that Google processes will be indexed.","quoteText":null,"sourceRole":"PRIMARY","verifiedAt":"2026-08-09T00:00:00","supportsText":"Three stages of crawling, indexing, and serving; JavaScript rendering; canonical selection; index stored across thousands of machines; indexing not guaranteed","domain":"developers.google.com","publisherName":"Google Search Central"},{"title":"Top ways to ensure your content performs well in Google's AI experiences on Search","url":"https://developers.google.com/search/blog/2025/05/succeeding-in-ai-search","excerpt":"Make sure your pages meet our technical requirements for Google Search, so that we can find them, crawl them, index them, and consider them for showing in our results.","quoteText":null,"sourceRole":"PRIMARY","verifiedAt":"2026-08-09T00:00:00","supportsText":"Technical requirements for AI features: Googlebot not blocked, HTTP 200, indexable content; preview controls; structured data must match visible content and enables rich results; higher-quality clicks from AI results","domain":"developers.google.com","publisherName":"Google Search Central Blog"},{"title":"OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User)","url":"https://developers.openai.com/api/docs/bots","excerpt":"Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.","quoteText":null,"sourceRole":"PRIMARY","verifiedAt":"2026-08-09T00:00:00","supportsText":"OAI-SearchBot surfaces sites in ChatGPT search; GPTBot collects training content; ChatGPT-User handles user-initiated actions and robots.txt may not apply; ~24-hour robots.txt propagation; blocked sites not shown in ChatGPT search answers","domain":"developers.openai.com","publisherName":"OpenAI"},{"title":"Web search","url":"https://developers.openai.com/api/docs/guides/tools-web-search","excerpt":"By default, the model's response will include inline citations for URLs found in the web search results.","quoteText":null,"sourceRole":"PRIMARY","verifiedAt":"2026-08-09T00:00:00","supportsText":"Model decides whether to search; non-reasoning, agentic, and deep research retrieval modes; deep research draws on hundreds of sources; inline citations by default; url_citation annotations with URL, title, and location; citations must be visible and clickable","domain":"developers.openai.com","publisherName":"OpenAI"},{"title":"Perplexity crawlers (PerplexityBot, Perplexity-User)","url":"https://docs.perplexity.ai/guides/bots","excerpt":"PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.","quoteText":null,"sourceRole":"PRIMARY","verifiedAt":"2026-08-09T00:00:00","supportsText":"PerplexityBot surfaces and links websites in results and is not used for foundation model training; Perplexity-User fetches pages for user requests and generally ignores robots.txt","domain":"docs.perplexity.ai","publisherName":"Perplexity"},{"title":"Does Anthropic crawl data from the web, and how can site owners block the crawler?","url":"https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler","excerpt":"Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses.","quoteText":null,"sourceRole":"PRIMARY","verifiedAt":"2026-08-09T00:00:00","supportsText":"ClaudeBot collects web content that may contribute to model training; Claude-User handles user-initiated requests; Claude-SearchBot analyzes content to improve search result relevance and accuracy; robots.txt controls","domain":"support.claude.com","publisherName":"Anthropic"},{"title":"Grounding with Google Search","url":"https://ai.google.dev/gemini-api/docs/google-search","excerpt":"The model analyzes the prompt and determines if a Google Search can improve the answer.","quoteText":null,"sourceRole":"PRIMARY","verifiedAt":"2026-08-09T00:00:00","supportsText":"Grounding definition; model analyzes the prompt and decides whether search improves the answer; automatically generates one or multiple queries; citation annotations with start and end indexes; reduces hallucinations by basing responses on real-world information","domain":"ai.google.dev","publisherName":"Google AI for Developers"},{"title":"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks","url":"https://arxiv.org/abs/2005.11401","excerpt":"RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.","quoteText":null,"sourceRole":"PRIMARY","verifiedAt":"2026-08-09T00:00:00","supportsText":"Origin of the RAG architecture; parametric memory (pre-trained model weights) combined with non-parametric memory (a dense document index accessed by a neural retriever); more specific and factual generation than parametric-only baselines","domain":"arxiv.org","publisherName":"arXiv (Lewis et al., NeurIPS 2020)"},{"title":"GEO: Generative Engine Optimization","url":"https://arxiv.org/abs/2311.09735","excerpt":"GEO can boost visibility by up to 40% in generative engine responses.","quoteText":null,"sourceRole":"INDEPENDENT","verifiedAt":"2026-08-09T00:00:00","supportsText":"Controlled benchmark of nine content strategies; quotation addition improved position-adjusted visibility about 41%, statistics and cite-sources about 30% each; keyword stuffing offered little to no improvement; effectiveness varies by domain; up to 40% overall visibility gain","domain":"arxiv.org","publisherName":"arXiv (Aggarwal et al., KDD 2024)"},{"title":"Ahrefs study: AI Overview citations vs. top-10 rankings (863,000 keywords, 4 million cited URLs)","url":"https://ahrefs.com/blog/ai-overview-citations-top-10/","excerpt":"at least in comparison to our initial study, it indicates that Google is selecting far fewer pages straight from the original SERP (~76% in July 2025 vs. ~38% today).","quoteText":null,"sourceRole":"INDEPENDENT","verifiedAt":"2026-08-09T00:00:00","supportsText":"March 2026 analysis of 863K SERPs and 4M AI Overview URLs; 37.9% of cited URLs in the first 10 blocks, 31.2% in positions 11 to 100, 31.0% beyond 100; shift from roughly 76% top-10 citations in July 2025; attributed to query fan-out","domain":"ahrefs.com","publisherName":"Ahrefs"},{"title":"The /llms.txt file","url":"https://llmstxt.org/","excerpt":"A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time.","quoteText":null,"sourceRole":"SUPPORTING","verifiedAt":"2026-08-09T00:00:00","supportsText":"llms.txt proposed September 3, 2024 as a standard file helping LLMs use a website at inference time; specification remains an open community proposal; no adoption claims by major platforms","domain":"llmstxt.org","publisherName":"llmstxt.org (Jeremy Howard)"}],"revisions":[],"relatedAnswers":[{"id":"de717b06-01be-4de5-9285-bec6bebad067","slug":"biggest-aeo-mistakes-to-avoid","question":"What are the biggest AEO mistakes to avoid?","publishedAt":"2026-08-28T14:15:04.474","confidenceScore":87,"confidenceLabel":"High","industry":{"id":"70ea3802-1fc6-4fd7-a505-140d38d1c74a","slug":"digital-marketing","label":"Digital Marketing","description":"SEO, content, demand gen, and growth marketing"},"topic":{"slug":"answer-engine-optimization","label":"Answer Engine Optimization","description":"How B2B teams get content selected and cited by AI answer engines.","schemaKind":null},"contributor":{"id":"ec39deab-44fe-48d8-9029-fefe993ab85a","slug":"answer-stack","displayName":"AnswerStack","websiteUrl":null},"snippet":"Seven high-cost AEO mistakes, each with the evidence for why it hurts and the specific correction: chasing referral volume instead of buyer fit, blocking AI crawlers by accident at the robots.txt or CDN layer, mass-producing thin optimized pages, ignoring the third-party sources AI answers actually cite, running AEO apart from crawlability and indexing, publishing with no measurement baseline, and treating a page that earned a citation as permanently cited.","url":"/q/biggest-aeo-mistakes-to-avoid"},{"id":"e99a45a2-0ef5-45e5-bec8-f6d813825f8c","slug":"what-offsite-signals-matter-for-aeo","question":"What off-site signals matter for AEO?","publishedAt":"2026-08-26T14:15:07.644","confidenceScore":76,"confidenceLabel":"Medium","industry":{"id":"70ea3802-1fc6-4fd7-a505-140d38d1c74a","slug":"digital-marketing","label":"Digital Marketing","description":"SEO, content, demand gen, and growth marketing"},"topic":{"slug":"answer-engine-optimization","label":"Answer Engine Optimization","description":"How B2B teams get content selected and cited by AI answer engines.","schemaKind":null},"contributor":{"id":"ec39deab-44fe-48d8-9029-fefe993ab85a","slug":"answer-stack","displayName":"AnswerStack","websiteUrl":null},"snippet":"Branded mentions, review corpus, third-party comparisons, community threads, earned media, and entity consistency are the six off-site signals with published evidence behind them. The quality of that evidence is uneven: a little is documented platform behavior, some is buyer survey data, and most is correlation from companies selling AI visibility tools. This breaks down what each signal is actually supported by, how long it takes to move, and the order most teams should work through them.","url":"/q/what-offsite-signals-matter-for-aeo"},{"id":"051baa33-8c61-42b3-8763-ea8a0267f827","slug":"what-makes-ai-engines-recommend-a-brand","question":"What makes AI engines recommend one brand over another?","publishedAt":"2026-08-24T14:15:05.494","confidenceScore":82,"confidenceLabel":"High","industry":{"id":"70ea3802-1fc6-4fd7-a505-140d38d1c74a","slug":"digital-marketing","label":"Digital Marketing","description":"SEO, content, demand gen, and growth marketing"},"topic":{"slug":"answer-engine-optimization","label":"Answer Engine Optimization","description":"How B2B teams get content selected and cited by AI answer engines.","schemaKind":null},"contributor":{"id":"ec39deab-44fe-48d8-9029-fefe993ab85a","slug":"answer-stack","displayName":"AnswerStack","websiteUrl":null},"snippet":"A cross-platform evidence map of why ChatGPT, Gemini, Perplexity, and Google's AI answers surface one brand instead of another. It separates what platforms actually document (index eligibility, crawler access, organic shopping results) from what correlational studies suggest (mention volume, listicle presence, reviews, rankings, freshness), with effect sizes drawn from studies covering 680 million citations, 75,000 brands, and 1.4 million prompts, plus the volatility data showing why these drivers shift month to month.","url":"/q/what-makes-ai-engines-recommend-a-brand"},{"id":"027226b0-7f49-42e5-924e-5a870eef73d9","slug":"how-far-aeo-extends-into-buyer-journey","question":"How far does AEO extend into the buyer journey?","publishedAt":"2026-08-21T14:15:05.932","confidenceScore":85,"confidenceLabel":"High","industry":{"id":"70ea3802-1fc6-4fd7-a505-140d38d1c74a","slug":"digital-marketing","label":"Digital Marketing","description":"SEO, content, demand gen, and growth marketing"},"topic":{"slug":"answer-engine-optimization","label":"Answer Engine Optimization","description":"How B2B teams get content selected and cited by AI answer engines.","schemaKind":null},"contributor":{"id":"ec39deab-44fe-48d8-9029-fefe993ab85a","slug":"answer-stack","displayName":"AnswerStack","websiteUrl":null},"snippet":"AI assistants now answer questions at every stage of a purchase, from problem framing to checkout and returns. This answer maps the evidence stage by stage: what buyers ask assistants at each point, which sources answer engines actually cite as the journey progresses, and the content each stage requires, using verified 2025-2026 data from Forrester, Bain, G2, Salesforce, Semrush, Ahrefs, xfunnel, and platform documentation from Google, OpenAI, and Stripe.","url":"/q/how-far-aeo-extends-into-buyer-journey"}],"contributorStats":{"verifiedAnswers":269,"openDisputes":0},"schemaJson":{"@context":"https://schema.org","@type":"Question","name":"How does answer engine optimization actually work?","text":"How does answer engine optimization actually work?","url":"https://www.answerstack.io/q/how-does-answer-engine-optimization-work","answerCount":1,"datePublished":"2026-08-09T20:58:23.191","author":{"@type":"Person","name":"AnswerStack Editorial Team","worksFor":{"@type":"Organization","name":"AnswerStack"},"url":"https://www.answerstack.io/contributors/answer-stack"},"about":[{"@type":"Thing","name":"Answer Engine Optimization"},{"@type":"Thing","name":"Digital Marketing"}],"acceptedAnswer":{"@type":"Answer","text":"Answer engine optimization works by aligning content with each stage of the pipeline AI answer engines run: crawlers discover and fetch pages [4][6], a search index or training corpus ingests them [2], a retrieval step pulls candidate passages when someone asks a question [1][9], the model grounds its generated answer in those passages [8], and a citation layer credits the small set of sources that shaped the response [5]. Platforms document the access and eligibility rules for the early stages: a page must be crawlable by the right bots, and on Google it must be indexed and snippet-eligible before it can appear as a supporting link [1][4]. The selection logic of the later stages is not published, so practitioners work from independent evidence, such as a controlled benchmark in which adding quotations, statistics, and cited sources raised visibility in generated answers by as much as 40% [10]. Effective AEO treats each stage as a filter and fixes the earliest failing stage first, because a page that never gets crawled or indexed cannot be retrieved, grounded on, or cited [1][2].","url":"https://www.answerstack.io/q/how-does-answer-engine-optimization-work","upvoteCount":0,"datePublished":"2026-08-09T20:58:23.191","dateModified":"2026-08-09T00:00:00","author":{"@type":"Person","name":"AnswerStack Editorial Team","worksFor":{"@type":"Organization","name":"AnswerStack"},"url":"https://www.answerstack.io/contributors/answer-stack"},"citation":[{"@type":"CreativeWork","name":"AI features and your website","url":"https://developers.google.com/search/docs/appearance/ai-features"},{"@type":"CreativeWork","name":"In-depth guide to how Google Search works","url":"https://developers.google.com/search/docs/fundamentals/how-search-works"},{"@type":"CreativeWork","name":"Top ways to ensure your content performs well in Google's AI experiences on Search","url":"https://developers.google.com/search/blog/2025/05/succeeding-in-ai-search"},{"@type":"CreativeWork","name":"OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User)","url":"https://developers.openai.com/api/docs/bots"},{"@type":"CreativeWork","name":"Web search","url":"https://developers.openai.com/api/docs/guides/tools-web-search"},{"@type":"CreativeWork","name":"Perplexity crawlers (PerplexityBot, Perplexity-User)","url":"https://docs.perplexity.ai/guides/bots"},{"@type":"CreativeWork","name":"Does Anthropic crawl data from the web, and how can site owners block the crawler?","url":"https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler"},{"@type":"CreativeWork","name":"Grounding with Google Search","url":"https://ai.google.dev/gemini-api/docs/google-search"},{"@type":"CreativeWork","name":"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks","url":"https://arxiv.org/abs/2005.11401"},{"@type":"CreativeWork","name":"GEO: Generative Engine Optimization","url":"https://arxiv.org/abs/2311.09735"},{"@type":"CreativeWork","name":"Ahrefs study: AI Overview citations vs. top-10 rankings (863,000 keywords, 4 million cited URLs)","url":"https://ahrefs.com/blog/ai-overview-citations-top-10/"},{"@type":"CreativeWork","name":"The /llms.txt file","url":"https://llmstxt.org/"}]}}}