Technical SEO for AI Crawlers: llms.txt, Schema and Rendering

Technical SEO for AI crawlers is the work of making sure AI systems can access, render and understand your website. If a crawler cannot reach or read your content, no amount of great writing will get you cited in ChatGPT, Perplexity or Google AI Mode. This guide covers the decisions that matter in 2026: robots.txt, rendering, llms.txt and schema.

What is technical SEO for AI crawlers?

It is the set of technical controls that determine whether AI-powered search and assistant systems can crawl, render and interpret your pages. It extends classic technical SEO to a wider set of bots with different capabilities.

• Access: robots.txt, CDN bot management and firewall rules.
• Rendering: whether content exists in the initial HTML or only after JavaScript runs.
• Understanding: semantic HTML, structured data and clear page structure.
• Hygiene: status codes, redirects, canonicals and sitemaps.

Which AI crawlers should you allow in robots.txt?

Allow the search and retrieval crawlers for any engine where you want to be cited, and make a separate business decision about training crawlers. Most AI companies now publish distinct user agents for each purpose.

These roles come from OpenAI's crawler documentation, Google's list of common crawlers and Perplexity's bot documentation. OpenAI notes it can take about 24 hours for robots.txt changes to take effect in its systems.

1. List every engine where you want visibility and the crawler each one uses.
2. Check robots.txt for blanket Disallow rules or wildcard blocks that catch AI bots.
3. Review CDN and WAF settings; one-click 'block AI bots' features often block search crawlers too.
4. Decide on training bots like GPTBot and Google-Extended as a separate content licensing choice.
5. Verify in server logs that the bots you allow are actually fetching pages successfully.

A sample robots.txt policy

A common 2026 policy for a business site is to allow search and retrieval while opting out of model training. In robots.txt that looks like separate groups for each user agent.

• User-agent: OAI-SearchBot with Allow: / so ChatGPT search can cite your pages.
• User-agent: PerplexityBot with Allow: / for Perplexity visibility.
• User-agent: GPTBot with Disallow: / to opt out of OpenAI model training.
• User-agent: Google-Extended with Disallow: / to opt out of Gemini training and grounding without affecting Google Search.
• Keep your normal Googlebot and Bingbot rules unchanged, and list your XML sitemap URL at the end of the file.

Remember that robots.txt is a request, not an enforcement mechanism. OpenAI and Google publish IP ranges for their crawlers, so you can verify that traffic claiming to be a given bot is genuine before allowing or blocking it at the firewall.

Do AI crawlers render JavaScript?

Many do not. If your content only appears after client-side JavaScript runs, several AI crawlers will see an empty or partial page.

Vercel's analysis of AI crawler traffic, published in December 2024 with MERJ, found that none of the major AI crawlers it studied, including those from OpenAI, Anthropic, Meta, ByteDance and Perplexity, rendered JavaScript. Gemini benefits from Googlebot's rendering infrastructure. Vercel also observed that ChatGPT's and Claude's crawlers hit 404 pages on roughly a third of their requests, compared with about 8% for Googlebot.

• Serve core content, headings, links and structured data in the initial HTML response.
• Use server-side rendering or static generation for marketing, product and content pages.
• Keep URLs stable and redirects tidy so AI crawlers do not waste requests on dead links.
• Test by fetching pages with JavaScript disabled or with curl to see what a non-rendering bot receives.

If your site is built with React, our guide to JavaScript SEO with SSR and SSG explains how to fix rendering without a full rebuild.

What is llms.txt and do you need one?

llms.txt is a proposed markdown file placed at a site's root that gives AI systems a concise summary of the site and links to its most useful content. It is optional, and no major search engine has said it uses the file as a ranking signal.

The llms.txt proposal was published by Jeremy Howard of Answer.AI in September 2024. Only an H1 with the site name is required; a summary blockquote and H2 sections of links are optional. The proposal also suggests offering clean markdown versions of pages at the same URL with .md appended.

Google's documentation states that you do not need new machine-readable files or AI text files to appear in AI Overviews or AI Mode. In May 2026, Google added an llms.txt check to Lighthouse's agentic browsing audits, which flags server errors but marks a missing file as not applicable.

Does schema markup help AI search?

Structured data helps machines understand what a page is about, but it is not a shortcut to AI citations. Google says no special schema is required for its AI features, and markup must match the visible content of the page.

The schema landscape is shifting. As reported by PPC Land, Google stopped showing FAQ rich results for all sites on May 7, 2026. Existing FAQPage markup causes no errors, and other engines may still read it, but it no longer earns a Google rich result.

• Prioritize Organization, Product, Article, LocalBusiness, Event and JobPosting where they genuinely apply.
• Use JSON-LD, which Google's structured data guide recommends as the easiest format to maintain.
• Validate with the Rich Results Test and monitor Search Console reports after deployment.
• Never mark up content users cannot see.

What does an AI crawler technical checklist look like?

Use this checklist quarterly or after any CDN, framework or hosting change.

1. Robots.txt allows Googlebot, Bingbot, OAI-SearchBot and PerplexityBot on public content.
2. CDN and firewall bot rules are reviewed and documented.
3. Key pages return full content in server HTML.
4. Status codes are accurate, with real 404s and minimal redirect chains.
5. XML sitemaps are current and submitted to Google and Bing.
6. Structured data is valid and matches visible content.
7. Server logs are monitored for AI crawler activity and errors.
8. llms.txt is published if your site is documentation-heavy.

How Advotis Agency can help

Advotis Agency audits crawler access, rendering and structured data so AI engines can read and cite your site. Explore our SEO services, or contact our team for a technical AI-readiness review.

Frequently asked questions

Advotis SEO Team

Written and reviewed by the SEO Team at Advotis Agency — specialists who build and grow websites, apps and AI products for clients every day. About us