llms.txt and AI Crawlers: What WordPress Site Owners Need to Set Up Now

llms.txt has become one of those things everyone tells you to add — it’s in the SEO newsletters, and plugins now generate one by default. Whether it deserves that attention is a separate question.

Here’s the honest answer: the evidence that llms.txt does anything measurable is weak. Three separate 2026 studies looked for a citation or ranking benefit and came up empty. That doesn’t mean skip it — it means you should know what it actually does before you spend the twenty minutes it takes to set one up.

In this piece, we’ll walk you through what llms.txt actually is and what the 2026 data says about whether AI crawlers even read it. Then, we’ll cover the part that matters even more: what actually controls whether ChatGPT, Claude, and the rest of them can reach your WordPress site at all. Robots.txt and your CDN do that job — llms.txt doesn’t.

In This Article:

What llms.txt Actually Is (and Isn’t) on WordPress

You’ve probably seen llms.txt described as “the new robots.txt for AI.” That name is exactly backwards, and getting confused about it is how people end up doing the wrong thing to their site.

llms.txt is a plain Markdown file that lives at yourdomain.com/llms.txt. It lists your most important pages with a one-line description of each, written to be read by an AI model instead of a search engine.

Robots.txt tells crawlers what they’re allowed to access. llms.txt does the opposite: it has zero restrictive power. Nothing stops an AI crawler from ignoring it completely, reading your entire site, and never once looking at the file.

It’s also not a standard. The original proposal came from developer Jeremy Howard, and no W3C committee, IETF working group, or AI company has ever agreed to support it.

Calling it “the AI robots.txt” is the single biggest source of confusion here — llms.txt can’t block, allow, or restrict anything. Think of it as a curated table of contents you hand an AI model, and hope it opens.

What the 2026 Data Says About Whether llms.txt Works

Before you spend real time curating one, it’s worth knowing what actually happens when you do — the data below runs against the assumption that llms.txt helps you get cited.

SE Ranking’s analysis of roughly 300,000 domains found that 10.13% had an llms.txt file as of November 2025. Adoption was flat across traffic tiers — high-traffic sites (100,001+ monthly visits) had the file at 8.27%, actually lower than mid-traffic sites at 10.54%.

Among the fifty most AI-cited domains in the study, exactly one had it. It isn’t an insider advantage.

(A separate measure tells a different story on the surface: HTTP Archive’s 2025 Web Almanac found just 2.13% desktop and 2.10% mobile adoption across 324,184 sites it checked.

That’s a roughly five-times gap from SE Ranking’s number, and it comes down to a different sample and a different detection method — not disagreement about the underlying trend. Either way, adoption is low.)

The more interesting finding is what happened when SE Ranking tried to use llms.txt to predict AI citations. Their model’s accuracy actually improved when they removed llms.txt as a variable — the file was adding noise, not signal, to whether a page got cited.

Crawler behavior backs this up. Monitoring across more than 500 million AI bot events over a 90-day window found only a few hundred requests aimed directly at /llms.txt. GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot overwhelmingly crawl your regular HTML pages instead of stopping to read the file you made for them.

None of that means llms.txt is a waste of time. It means the case for adding one has nothing to do with getting cited by ChatGPT — and the real case, covered next, is a better reason to actually do it.

Three Honest Reasons to Add llms.txt to WordPress Anyway

If citations and rankings aren’t the payoff, here’s what actually is:

• It’s an asymmetric bet. Twenty minutes of setup buys you a position if agentic AI routing ever standardizes around files like this one. If it doesn’t, you’ve lost twenty minutes — not a strategy.
• It forces a useful audit. Writing a one-sentence description of your 40 most important pages is how you notice the duplicate posts, the orphaned pages, and the ones that don’t actually say anything. Several teams get more value out of that exercise than out of the file it produces — it’s the same instinct behind getting your schema markup in WordPress right: the audit matters as much as the markup.
• It’s a surface for the next kind of visitor. The more interesting case isn’t a search citation — it’s an AI agent trying to work out what your business sells and which page is the authoritative answer for each thing it’s asked about.

None of these reasons requires you to believe llms.txt will move a single ranking. They’re reasons to spend twenty minutes on it, not reasons to build a content strategy around it.

How to Add llms.txt to WordPress (4 Methods)

If you’ve decided the twenty minutes is worth it, here’s the fastest path based on what’s already running on your site — easiest first.

1. AIOSEO: Add llms.txt to WordPress Without Touching Code

If you already run All in One SEO, check before you build anything — you may already have a file. AIOSEO generates llms.txt by default, and per the 2025 Web Almanac, roughly 40% of the llms.txt files published across the web trace back to this one plugin.

The researchers were careful to note they can’t tell whether every site owner chose that deliberately or simply never noticed — which is exactly why it’s worth opening yours.

Open Sitemaps in your AIOSEO menu, then click the LLMs.txt tab. You’ll likely find two toggles already switched on: Enable llms.txt and Enable llms-full.txt, each capped by default at 1,000 URLs per post type or taxonomy.

That cap is generous enough to turn your file into a sitemap with extra steps, so this is exactly the spot to curate rather than accept what’s there.

A separate “Convert Posts to Markdown” toggle sits nearby and ships off by default — leave it off unless you’ve thought through the duplicate-content tradeoff covered in the next section.

Five minutes there, and you’ve either confirmed your existing file is fine or found out it needed curating. Either way, you’re done with this method.

2. Yoast SEO: Enable llms.txt on WordPress

If you run Yoast SEO instead, go to Settings, then Site Features, and look for AI tools. Turn on LLMS.txt, then click “Customize llms.txt file” to hand-pick which pages make the list rather than accept Yoast’s automatic selection.

The feature shipped free in version 25.3 in June 2025 — one caveat worth knowing before you start: it isn’t available on multisite installs.

3. Rank Math: Add the LLMs Txt Module in WordPress

Rank Math keeps this behind a module you switch on first. Open the Rank Math Dashboard, enable LLMS Txt, then head to General Settings and click Edit llms.txt.

Worth flagging: Rank Math’s default cap is 100 links per post type, well past the 10-to-40 curated range that actually makes a file worth reading — plan to trim it down rather than publish what it generates.

4. Manual Upload: Add llms.txt to WordPress Yourself

Prefer to skip the plugin route entirely? Write the file yourself in a plain-text editor, upload it to your site’s root folder — the same place as public_html — using your host’s file manager or an FTP client, then confirm it loaded correctly by visiting yourdomain.com/llms.txt directly in your browser.

A handful of managed hosts are starting to build this in as well — Hostinger’s LLM Optimization panel is one example — though for most WordPress site owners, the SEO plugin you already have installed is the simpler path.

How to Write an llms.txt File for WordPress That Isn’t Junk

Whichever method you used above, the file a plugin generates for you and the file worth having are two different things.

Default-on generation is exactly how roughly 40% of the web’s llms.txt files ended up published without anyone deciding what belongs in them — and a file that dumps every URL on your site isn’t a curated guide, it’s a sitemap with extra steps.

The format itself is simple: an H1 with your site name, a one- or two-sentence blockquote summary of what your business does, optional prose beneath it, then H2 sections grouping links with a one-line description of each. Here’s a template you can copy and adapt directly:

# Your Business Name

> One or two sentences summarizing what your business does and who it serves.

## About
- [About Us](https://yourdomain.com/about/): One-sentence description of what this page covers.
- [Contact](https://yourdomain.com/contact/): One-sentence description.

## Guides
- [Guide Title](https://yourdomain.com/guide-slug/): One-sentence description of what the reader learns.

## Products or Services
- [Product Name](https://yourdomain.com/product-slug/): One-sentence description of what it does.

Last reviewed: August 2026

A few rules of thumb make the difference between a useful file and a wasted one:

• It should be served as plain text at exactly /llms.txt — not a redirect, not a subfolder.
• Absolute URLs (https://yourdomain.com/page/) work better than relative ones.
• 10 to 40 curated pages is the sweet spot — not every page you’ve ever published.
• A last-reviewed date is worth including, and actually revisiting — a stale file is worse than none.

One thing worth skipping: llms-full.txt Markdown mirrors of every page, unless you’re prepared to handle the indexation problem that comes with them. Publishing a full Markdown copy of every page creates duplicate content at scale, dilutes crawl budget, and has been known to outrank the original page it copied.

If you do publish one anyway, keep it out of your sitemap and consider a noindex tag on it.

What Actually Controls AI Crawler Access on WordPress: Robots.txt

This is where the real control lives. llms.txt can only suggest. Robots.txt can actually say no — and every major AI crawler except one well-documented exception honors it.

The real decision isn’t “AI in or out.” It’s which of four outcomes you want for your content: used for training, cited in AI answers, both, or neither. Robots.txt is how you make that call crawler by crawler, because different AI bots do genuinely different jobs.

AI Crawlers on WordPress: The User-Agent Table

Here’s how the crawlers you’ll see in your robots.txt file and your access logs break down by what they actually do:

User-Agent	Operator	What It Does	Respects robots.txt?
GPTBot	OpenAI	Crawls pages to train models	Yes
ClaudeBot	Anthropic	Crawls pages to train models	Yes
CCBot	Common Crawl	Builds the shared dataset several AI labs train on	Yes
Meta-ExternalAgent	Meta	Crawls pages to train models and products	Yes
Bytespider	ByteDance	Crawls pages to train models	No — widely documented ignoring it
OAI-SearchBot	OpenAI	Crawls pages for live citations in ChatGPT search	Yes
Claude-SearchBot	Anthropic	Crawls pages for live search results in Claude	Yes
PerplexityBot	Perplexity	Crawls pages for live search results	Yes
ChatGPT-User	OpenAI	Fetches one page a user asked ChatGPT about, in real time	OpenAI says rules “may not apply”
Claude-User	Anthropic	Fetches one page a user asked Claude about, in real time	Yes
Perplexity-User	Perplexity	Fetches one page a user asked about, in real time	Perplexity says rules “may not apply”
Google-Extended	Google	Opt-out token only — never fetches a page	Not a crawler; no HTTP request to respect
Applebot-Extended	Apple	Opt-out token only — never fetches a page	Not a crawler; no HTTP request to respect

Two rows deserve a closer look, since they’re the most misunderstood tokens on the list. Google-Extended doesn’t touch your Google Search or AI Overviews visibility at all — it only opts you out of Gemini training and grounding and Vertex AI grounding. Applebot-Extended is narrower still: it opts your content out of Apple Intelligence training only, while regular Applebot keeps crawling exactly as before.

Compliance isn’t a blanket rule, either — it’s per vendor. Bytespider is widely documented ignoring robots.txt outright, and OpenAI and Perplexity both say rules “may not apply” to their user-triggered fetchers, ChatGPT-User and Perplexity-User, because a person requested the fetch.

Anthropic documents the opposite for its own bots — all three, including Claude-User, honor robots.txt without exception.

The Canonical robots.txt Pattern for AI Crawlers on WordPress

The split most sites land on — and the one OpenAI’s own documentation endorses — is allowing the crawlers that power live AI search results while blocking the ones that train on your content. Here’s that pattern, ready to paste into your robots.txt file:

# Allow AI search crawlers (citations in AI answers)
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Which crawlers you block comes down to the decision you made above — some site owners are comfortable letting training crawlers in too, in exchange for wider exposure. There’s no universally correct answer here, only the one that matches what you actually want done with your content.

How to Edit robots.txt on WordPress

If you run AIOSEO, open Tools, then Robots.txt Editor, and flip on “Enable Custom Robots.txt” first — it ships off by default. The same screen has a native “Block AI Crawlers” checkbox panel (All AI Crawlers, Google AdsBot, OpenAI GPTBot, Common Crawl CCBot, Google Gemini & Vertex AI Bots — all unchecked by default).

It’s a real, fast shortcut, but a blunt one: no ClaudeBot or PerplexityBot entry, and no way to allow search crawlers while blocking training ones. For the pattern above, use the manual “Add Rule” table earlier on the same screen instead.

Yoast and Rank Math each have their own robots.txt editing tools under their respective settings menus if you’re running one of those instead.

And if you’d rather not touch a dedicated SEO plugin’s robots.txt tool at all, WPCode’s snippet library includes a robots.txt editor that works the same way.

Your CDN Can Override llms.txt and Robots.txt: The Cloudflare Change Landing September 15

Block AI training crawlers through your CDN, and you might take out your own Google crawl access as collateral damage.

Googlebot, Applebot, and Bingbot are multi-purpose — the same user-agent handles regular search crawling and AI-training crawling at once. A CDN policy written to say “no AI training” can also say “no Googlebot” without anyone noticing until traffic drops.

On July 1, 2026, Cloudflare replaced its single “Block AI bots” toggle with three independent controls: Search, Agent, and Training. Every plan gets them, including Free. Each one can be set to block on all pages, block only on pages that display ads, or not block at all.

On September 15, 2026, new domains onboarding to Cloudflare start out with Training and Agent blocked on ad-supporting pages by default, while Search stays allowed. If your site is already on Cloudflare, that new default doesn’t apply to you automatically — you can opt into it, or explicitly opt out, any time before that date.

This is separate from Bot Fight Mode — a different Cloudflare feature entirely, opt-in and built for generic bot abuse, not the AI-specific Search, Agent, and Training controls described here.

This one screen has more control over whether AI platforms can reach your content than everything else in this post combined — which is really the whole thesis in one setting.

Last updated: August 2026.

How to Know If Your llms.txt and robots.txt Changes Are Working

You can do everything above correctly and still have no idea whether any of it changed anything. That’s the gap this section closes.

On the server side, you can filter your access logs for requests to /llms.txt from known AI user-agents, or set a honeypot: add a URL that appears nowhere else on your site and watch for a hit. Cloudflare’s own bot analytics can show you this without digging through raw logs. None of it will ever catch Google-Extended or Applebot-Extended, though — they’re opt-out tokens, not crawlers, so they never make an HTTP request for anything to log.

But crawler fetches are a vanity metric. Whether any of this sends you real traffic that converts is the outcome worth checking.

Default GA4 reporting isn’t built to answer that cleanly. Google’s own AI Overviews and AI Mode are excluded from the AI Assistant channel and get classified as plain Organic Search instead.

That channel also only counts a session as AI-driven when the referrer matches a known list of AI assistants. Any AI referral session that arrives without a referrer falls into Direct traffic instead — invisible as AI traffic even though that’s exactly what it was. The channel itself is new, too; it shipped May 13, 2026, so there isn’t much historical data to lean on yet.

That’s exactly the gap MonsterInsights — the WordPress analytics plugin — built its AI Traffic Report to close (available on MonsterInsights Pro and above). Head to Traffic, then AI Traffic, inside your WordPress dashboard.

You’ll see sessions, engaged sessions, purchases, and revenue broken out by platform — ChatGPT, Claude, Gemini, and Poe. A compare-to-previous view is built in, so you can put an actual number on before and after.

Worth being precise about what this measures, because the distinction matters: it’s referral traffic, not crawler fetches, not hits to /llms.txt, and not your CDN’s bot policy.

Those live in the server logs and the Cloudflare dashboard covered above — this report shows you the business result those settings are supposed to produce, not the settings themselves.

One more small thing worth doing first: log a Site Note, under Insights, then Site Notes, marking the date you shipped your llms.txt and robots.txt changes. It’s available starting on the Plus plan.

That gives the AI Traffic Report’s before-and-after comparison an actual fixed point to measure against, instead of a vague memory of “sometime last month.”

Put those two together and you’ve built an actual experiment: ship the changes, mark the date, then check your AI traffic 30 and 90 days out and see what moved.

Your WordPress AI Crawler Priority Order Before September 15

If you only have time for some of this, do it in this order — the order is the argument.

1. Check your CDN. Read your Cloudflare Search, Agent, and Training settings before September 15 — this is the one item here with a hard deadline and real consequences.
2. Get your crawl access right. Set up robots.txt with the allow-search, block-training pattern above.
3. Know your baseline. Check your AI traffic before you change anything, so you have something real to compare against later.
4. Work on your content. Clear answer structure and original data do more for AI visibility than any file will.
5. Add llms.txt. It’s worth twenty minutes. It’s not worth a strategy.

The first three are worth doing this week — the rest of this post has already done its job.

FAQs About llms.txt WordPress

Your robots.txt file and your CDN settings are doing the real work here — llms.txt is worth the twenty minutes, nothing more. Here are a few more guides worth a read:

• AI Search Engine Optimization: The Complete Ranking Guide
• Schema Markup in WordPress: Let Me Simplify It for You
• How to Configure Your WordPress Website With Google Search Console
• What Is Direct Traffic in Google Analytics
• Google Ranking Factors

Follow us on Twitter, Facebook, and YouTube for more Google Analytics tips.