AEO Website Optimization, How to Build a Site AI Answers Can Cite
Anurag Kurmi
Senior Full Stack Engineer, Unico Connect
In this article
- Quick Answer
- Key Takeaways
- What AEO Website Optimization Means in 2026
- Who Builds What on an AEO Website
- Can AI Crawlers Execute JavaScript?
- AI Crawlers and robots.txt, Search vs Training vs User Fetch
- Structured Data for AI Search, What It Does and Does Not Do
- An Answer First Page Template
- Entity Signals, Sitemaps and IndexNow
- AEO Myths vs Evidence
- How to Measure AI Search Visibility
- AEO Website Optimization Checklist for Developers
- Building an AEO Ready Website With Unico Connect
- Frequently Asked Questions
AEO website optimization starts with a boring test. Ask your server for a page the way a bot would, then read the raw reply. An empty div and a script tag? In most published tests, that left the fetchers behind ChatGPT, Claude and Perplexity with nothing to quote. We build websites, so weigh our view with that in mind. Every vendor statement and study below was checked at its source on October 29, 2026.
Quick Answer
AEO website optimization, or answer engine optimization for a website, is the build and content work that makes your pages easier for AI answer engines such as Google AI Overviews, ChatGPT and Perplexity to read and cite. Google says AI Overviews and AI Mode need no extra files, markup or special schema. A page has to be indexed, eligible for a snippet and not excluded in Search Console. So AEO website optimization is mostly careful SEO. The developer half is the full text in the first HTML response, which is all most AI fetchers read in published tests, plus robots.txt and CDN rules that let search crawlers in and an answer block in every template. The marketing half is answer first copy and consistent company facts. Nobody can guarantee a citation, and llms.txt does nothing for Google Search.
Key Takeaways
- Rendering is the biggest developer decision. Headings, answers, prices and JSON-LD belong in the server HTML.
- OpenAI and Anthropic run separate search, training and user fetch bots, so each robots.txt token is its own call.
- Structured data helps search engines read a page. It is not an AI citation switch.
- Search Console, Bing Webmaster Tools and GA4 show AI visibility, but we found no citation report for site owners from OpenAI, Anthropic or Perplexity.
What AEO Website Optimization Means in 2026
Answer engine optimization (AEO) and generative engine optimization (GEO) are two names for one job, visibility in AI search experiences. Google defines both terms and says optimizing for generative AI search "is optimizing for the search experience, and thus still SEO." The Bing Webmaster Guidelines open with "SEO Fundamentals Still Apply to Grounding and AI Experiences."
Google grounds AI Overviews and AI Mode in pages retrieved from its Search index, meaning it builds answers from them. ChatGPT search sometimes sends rewritten queries to partner search providers. Anthropic lists Brave Search as a web search subprocessor for Claude, and Brave skips pages that Googlebot may not crawl. So for Google, ChatGPT and Claude alike, the classic crawl and index basics carry over.
Voice search is a separate job, covered in voice search optimization for web apps.
Who Builds What on an AEO Website
Developers decide whether a page can be fetched and parsed. The content team decides what it says, and Google rates that part higher. Its AI optimization guide says content people find "unique, compelling, and useful" will likely influence your presence in generative AI search "more than any of the other suggestions in this guide."
Table 1. AEO work split between developers and marketing
| Layer | The development team builds | Content and marketing own |
|---|---|---|
| Crawl access | robots.txt by bot purpose, CDN rules, status codes | The decision on AI training |
| Rendering | Server rendered or static HTML for every public page | Key facts in text, not images or PDFs |
| Page structure | Semantic headings, real HTML tables, an answer block in each template | Answer first copy, sections that stand alone |
| Structured data | JSON-LD generated from the same data as the page | Accurate author bios, dates and company facts |
| Entities | One Organization and author identity across the site | The same company name and facts on every profile |
| Discovery | Sitemap with honest lastmod, IndexNow on deploy, canonical tags | Updating pages when facts change |
| Speed | Core Web Vitals budgets in the build | Image and video choices |
| Measurement | AI referrer reporting and bot logs | Monthly buyer question checks in AI assistants |
Can AI Crawlers Execute JavaScript?
Mostly not. Googlebot does, for the Search index that AI Overviews and AI Mode draw on. In published tests, the live fetchers behind ChatGPT, Claude and Perplexity read only the HTML the server sent, while results for Gemini and Copilot vary by test. The Google JavaScript SEO guide still recommends server side or pre rendering because "not all bots can run JavaScript," and the Claude web fetch tool docs say it "currently does not support websites dynamically rendered with JavaScript."
A December 2024 vendor study by Vercel, which maintains Next.js, and MERJ found that ChatGPT and Claude crawlers fetched JavaScript files but did not execute them. In a one site test of pasted URLs in June 2026, Search Engine World hid a real number behind a script, and ChatGPT, Claude, Gemini, Perplexity, Meta AI and Copilot all reported the decoy in the HTML. When Glenn Gabe of GSQi retested in October 2026, ChatGPT, Claude and Perplexity still missed text drawn by a script, while Gemini read it in all three of his cases and Copilot in two.
Server logs tell a different story for OpenAI. A developer report from September 30, 2026 and a one site log study from October 22, 2026 saw OpenAI crawlers render pages on some sites, which OpenAI has not documented. The log study also saw the ChatGPT-User fetcher read HTML only.
How a page renders is the decision that most affects traffic, because search engines and AI answer engines read the HTML a page returns. Our guide to choosing between Next.js and React covers the framework side. In a September 2026 log study, Forge AI Bot Log, a WordPress bot log plugin vendor, lists load more buttons, infinite scroll, script drawn charts and click to answer calculators as invisible to HTML only bots.
We ran the test below on one of our own posts on October 29, 2026. With the GPTBot, ClaudeBot and PerplexityBot user agents, the HTML that came back already held the headline, every section heading, the full article text and four blocks of structured data.
See Your Page the Way a Crawler Does
Fetch the raw HTML with each user agent
Fetch the page as OAI-SearchBot and print the status code.
curl -sL -A OAI-SearchBot \
-w "%{http_code}\n" \
-o oai.html \
https://example.com/page
Repeat with ClaudeBot, PerplexityBot and Googlebot, saving each to its own file. Search each file for the H1, the answer paragraph, prices and application/ld+json.
Compare the source with the rendered page
Open view source, then the Elements panel in Chrome DevTools. Text found only in Elements was added by JavaScript, and a fetcher that skips scripts never sees it.
Check status codes and blocks
Each bot should get HTTP 200, not a 403 or a challenge page. A firewall that checks IP ranges may treat your test differently, so confirm real bot hits in your logs.
Validate the JSON-LD
Run the page through the Rich Results Test and confirm the markup matches the visible text on the page.
Inspect the page in Search Console
URL Inspection shows the HTML Googlebot rendered and whether the page is indexed.
AI Crawlers and robots.txt, Search vs Training vs User Fetch
OpenAI and Anthropic now run separate bots for search, training and pages a person asks about, and Perplexity splits search from user fetches. Allow the search bots if you want to be cited, and treat training bots as a licensing choice. User fetchers may ignore robots.txt, so it is not access control.
Table 2. AI crawlers by purpose, checked October 29, 2026
| robots.txt token | Operator and purpose | If you block it |
|---|---|---|
| Googlebot | Google Search, including AI Overviews and AI Mode | No crawling for Search or its AI features |
| Google-Extended | Google, a token for Gemini training and grounding, no crawler of its own | No Gemini training or grounding use, Search inclusion unaffected |
| Bingbot | Microsoft, Bing search, Copilot and grounding results | No crawling for Bing or Copilot |
| OAI-SearchBot | OpenAI, ChatGPT search | Left out of ChatGPT search answers, except navigational links |
| GPTBot | OpenAI, model training | Content marked as not for training |
| ChatGPT-User | OpenAI, fetches for a ChatGPT user | OpenAI says robots.txt rules may not apply |
| Claude-SearchBot | Anthropic, Claude search results | Anthropic says search visibility may drop |
| ClaudeBot | Anthropic, model training | Future content marked as not for training |
| Claude-User | Anthropic, fetches for a Claude user | No fetch for that user |
| PerplexityBot | Perplexity search, not training | Perplexity recommends allowing it for search results |
| Perplexity-User | Perplexity, fetches for a Perplexity user | Perplexity says it generally ignores robots.txt |
Rows come from the vendor page for OpenAI, Anthropic, Perplexity, Google or Bing. This example lets the search bots in and opts out of OpenAI, Anthropic and Gemini training. Other training tokens, such as Applebot-Extended for Apple foundation models, need a group of their own if you want them out too.
# Search and answer engines
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /admin/
Disallow: /search/
# Training bots, opted out
# Google-Extended also covers
# grounding in Gemini Apps
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
# Everyone else
User-agent: *
Disallow: /admin/
Disallow: /search/
Sitemap: https://www.example.com/sitemap.xml
A bot with its own group ignores the User-agent: * group, because Google documents that named groups and the global group are not combined. Repeat shared Disallow lines in each named group, or a GPTBot group with only an Allow line lets GPTBot skip your admin rules. OpenAI says changes can take about 24 hours to reach ChatGPT search, and Perplexity says up to 24 hours.
CDN Rules and the Search Generative AI Control
Google lists crawling allowed "by any CDN or hosting infrastructure" among its AI feature basics, and OpenAI asks sites to allow its published OAI-SearchBot IP ranges. Cloudflare documented new defaults for new domains from September 15, 2026 that block Training and Agent bots on pages that show ads. Crawlers that combine search and training fall under the training block too, while Search only bots stay allowed. Ask whoever runs your CDN which bot categories it blocks today.
As of August 31, 2026, Google has rolled out a Search generative AI control to all sites, under Settings in Search Console, and a site set to Exclude drops out of AI Overviews, AI Mode and AI features in Discover. Include is the default for a property with no parent. Check snippet rules too, since Google says nosnippet keeps a page from being a direct input for AI Overviews and AI Mode, and Bing says NOARCHIVE keeps it out of Copilot answers.
Structured Data for AI Search, What It Does and Does Not Do
JSON-LD schema markup is worth shipping for classic search. Neither Google nor Bing promises it earns AI citations. Google says there is "no special schema.org markup you need to add" for generative AI search, and still recommends structured data that matches the visible text for rich result eligibility. Bing says structured data "may support clearer grounding but does not guarantee visibility or grounding traffic."
Ahrefs, which sells AI visibility tracking, ran a vendor study of 1,885 pages already cited in AI Overviews that added JSON-LD between August 2025 and March 2026. AI Mode and ChatGPT citations did not measurably change. AI Overview citations fell 4.6 percent against matched pages, a gap Ahrefs says it cannot definitively pin on schema.
Organization markup on the home page can help Google tell your company apart and pick your logo. For articles, add BlogPosting with a link to the author page and a BreadcrumbList, as each bylined post on this blog does.
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Organization",
"@id": "https://www.example.com/#organization",
"name": "Example Co",
"url": "https://www.example.com",
"logo": "https://www.example.com/logo.png",
"description": "Example Co builds payroll software for US staffing agencies.",
"sameAs": [
"https://www.linkedin.com/company/example-co",
"https://www.crunchbase.com/organization/example-co"
]
}
</script>
An Answer First Page Template
Answer first is something you build into the template, not a tip you hand writers. Each section opens with a one or two sentence answer to its heading, and the detail comes after. Bing asks for essential information near the top of the page and content that stands on its own. Microsoft advises against hiding important answers in tabs or expandable menus, because AI systems may not render hidden content. Redesigns love accordions. Check them twice.
The rest is plumbing. Put data in real HTML tables and keep the author and date on the page. Googlebot fetches the first 2MB of a page, headers included, and ignores anything after that, so the title, canonical tag and JSON-LD go near the top of the HTML. On our own site the scroll animations hide text only after a script runs. A crawler that skips JavaScript still gets every word.
Entity Signals, Sitemaps and IndexNow
An entity is the company, person or product a page is about. Bing asks for clear and consistent naming of people, organizations, products and locations, and says it improves citation accuracy. This part is admin work, and nobody enjoys it. You want one Organization identity with a stable @id, an author page for each writer and the same company facts on every directory. If another business shares your name, say so in your Organization markup. Ours does, for the name Unico.
Sitemaps are where static builds quietly go wrong. Google uses sitemap lastmod only when it is consistently accurate, so set it from real edit dates, never the build time. IndexNow tells participating engines such as Bing when a URL is added, updated or deleted, and each one shares the submission with the others. Google is not a participant. Bing prefers streamed submissions to batches, so our production builds send only what changed, new and deleted pages included.
AEO Myths vs Evidence
Several AEO tactics sold today fail a check against the source documents.
What Google says its Search does not need
- llms.txt files, other AI text files or Markdown copies of pages. Google Search ignores them, and they neither harm nor help.
- A page for every way people phrase a query, which Google treats as scaled content abuse when the aim is to manipulate AI answers.
- Inauthentic mentions seeded across the web.
97%
of llms.txt files on sites using Ahrefs Web Analytics received no requests in May 2026
Ahrefs vendor study, June 15, 2026
Table 3. Common AEO claims checked against the sources
| Claim | What the sources say |
|---|---|
| An llms.txt file gets you cited | Lighthouse marks a missing file not applicable, and the llms.txt spec says the files are used most heavily for software documentation that coding agents follow |
| FAQ or HowTo schema gets you into AI answers | Google stopped showing HowTo rich results in 2023 and FAQ rich results on May 7, 2026. Bing says FAQ sections help AI systems reference a page, and its guidelines say structured data does not guarantee grounding |
| Blocking Google-Extended keeps you out of AI Overviews | Google-Extended does not affect Search inclusion. Use the Search generative AI control or nosnippet |
| An AEO tool can see how Google AI ranks you | Google says no outside tool has access to its internal ranking or AI systems |
How to Measure AI Search Visibility
No single report covers every assistant, so you need four sources.
Table 4. Where AI visibility shows up
| Source | What it shows | What it does not show |
|---|---|---|
| Search Console Generative AI report | Impressions in AI Overviews and AI Mode by page, country, device and date, for all sites since August 31, 2026 | Queries or clicks, which are mixed into the main Performance report under Web |
| Bing AI Performance | Citations in Copilot, Bing AI summaries and select partner integrations, cited pages and a sample of grounding queries, since February 2026 | Clicks, or which other sites were cited |
| GA4 AI Assistant channel | Visits referred by assistants such as ChatGPT, Gemini, Copilot and Claude, since May 13, 2026 | AI Overviews and AI Mode, which stay in Organic Search |
| Server logs | Requests from each bot, checked against its published IP ranges | Whether a fetch became a citation |
Our own report is a fair example of what you get. Over the three months to October 27, 2026, Search Console counted about 222,000 impressions for unicoconnect.com in AI Overviews and AI Mode, spread across 396 pages. It could not tell us which queries triggered them or whether anyone clicked.
GA4 has a gap too. Google does not name Perplexity in its AI Assistant channel definition, so check which channel your perplexity.ai visits land in. And from OpenAI, Anthropic and Perplexity, we found no citation report for site owners on October 29, 2026. Until one exists, ask each assistant the same buyer questions every month, logged out and more than once, and write down which sites it cites.
AEO Website Optimization Checklist for Developers
Use this as an AEO audit checklist before launch or in the brief for a rebuild.
- Public pages return HTTP 200 with text, headings and JSON-LD in the first HTML response
- robots.txt allows the search bots and repeats shared rules in each named group
- CDN rules let verified search bots through, and the Search generative AI control is set to Include
- No stray noindex, nosnippet, NOARCHIVE or NOCACHE on pages meant to be cited
- JSON-LD is generated from page data and passes the Rich Results Test
- One Organization @id and an author page per writer, with the same company facts on every profile
- Templates put the answer under the heading and show the author and date
- Sitemap lastmod uses real edit dates, and IndexNow fires on deploy
- Core Web Vitals pass at the 75th percentile, LCP within 2.5 seconds, INP at 200 milliseconds or less, CLS at 0.1 or less
- Buttons, links and labels are real HTML elements, since browser agents read the DOM and accessibility tree
- Search Console and Bing Webmaster Tools are verified, and analytics shows which channel perplexity.ai visits land in
Building an AEO Ready Website With Unico Connect
Unico Connect builds websites and web apps, and this site serves prerendered Next.js pages with JSON-LD. On our four hour marketing site build, SEO and AEO were built in from the first pass. Semantic markup, metadata, structured data, and answer engine ready phrasing went in while the site was being built.
Frequently Asked Questions
Is AEO different from SEO?
Only at the edges, because Google calls AEO still SEO and Bing says SEO fundamentals still apply. What is new? AI crawler tokens to decide on, a Search Console switch for AI features, reports on AI impressions and citations, and Bing guidance to answer near the top of the page.
How do I get my website cited by ChatGPT?
Allow OAI-SearchBot in robots.txt and let its published IP ranges through your CDN, because OpenAI says sites that block it are not shown in ChatGPT search answers, except as navigational links. Then it is ordinary work, a server rendered, indexed page that answers the question plainly. OpenAI says placement is not guaranteed.
How do I show up in Google AI Overviews?
There is no separate submission. Google says a page must be indexed and eligible to show with a snippet, with no extra technical requirements, and the site must not be excluded by the Search generative AI control. Plenty of queries never get an Overview, because Google shows one only when it judges it additive to classic Search.
Can ChatGPT read a JavaScript website?
Usually not when it opens a page for a user. In published tests through October 2026, ChatGPT, Claude and Perplexity read only the HTML the server sent, though two log reports from September and October 2026 saw OpenAI crawlers render pages on some sites. Googlebot renders JavaScript for the Search index behind AI Overviews and AI Mode. Put the content that matters in the server HTML either way.
Should my website have an llms.txt file?
Treat an llms.txt file as optional. Google Search ignores llms.txt, and an Ahrefs vendor study of 137,210 sites that use Ahrefs Web Analytics found that 97 percent of the llms.txt files on those sites received no requests in May 2026.
Can we block AI training without leaving AI search?
Mostly, by allowing OAI-SearchBot, Claude-SearchBot and PerplexityBot and blocking GPTBot and ClaudeBot. OpenAI keeps its settings independent, and PerplexityBot is not used for training. Google-Extended is the hard call, because blocking it also stops grounding in Gemini Apps, though Google says it does not change Google Search inclusion or ranking.
