llms.txt - What It Is, What It Does, and How to Ship One

llms.txt: what it is, and what it is not (yet)

Few emerging standards have collected hype this fast. Here is the honest version: what the file is, what evidence exists, and why we ship it anyway - across every property we run.

In short

llms.txt is an emerging convention: a markdown file at your site root that gives AI systems a curated guide to your most useful content. It costs minutes to ship and cannot hurt; claims that it boosts rankings today are unproven. We ship it across our own estate and say exactly that.

What the file is

A proposed convention: a markdown file at /llms.txt that describes your site for LLM consumption - what the site is, and a curated list of your most useful pages, optionally with an expanded /llms-full.txt carrying the content itself. Think of it as a sitemap written for readers that read prose: instead of thousands of undifferentiated URLs, a short, prioritized guide.

What it does today - honestly

Confirmed: it is fetched. Our own logs show AI bots requesting llms.txt and llms-full.txt from our properties.
Plausible, unproven: that it influences which of your pages get cited. Adoption by engines is uneven and mostly undocumented; anyone claiming a measured ranking effect should show the measurement.
False: that it is a ranking requirement, a legal signal, or a replacement for robots.txt. It is none of those.

Why we ship it anyway

The cost is minutes; the failure mode is nothing; the upside - if the convention consolidates - accrues to early, clean implementations. That asymmetry is the whole argument, and it is enough. Every Webclat property ships llms.txt, several with full markdown mirrors of key pages, generated automatically at build time so they never drift stale.

How to ship one correctly

Write it as a guide, not a dump: one line on what the site is, then your genuinely most useful pages with one-line descriptions each.
Curate hard - twenty strong links beat four hundred indiscriminate ones. You are telling a reader with limited attention where the good parts are.
Keep it generated, not hand-maintained - wire it into the build so new content flows in and dead links flow out.
Pair it with the fundamentals it does not replace: clean structure, quotable answers, a deliberate crawler policy.

Common questions

Does llms.txt improve AI visibility?

Unproven. It is fetched - our logs confirm that - but a measured citation effect has not been credibly demonstrated by anyone, including us. We ship it for the cost/upside asymmetry and label the rest honestly.

Is llms.txt a standard?

A proposed convention with growing but uneven adoption - not a ratified standard. Treat current implementations as low-cost positioning, not compliance.

llms.txt vs robots.txt - what is the difference?

robots.txt controls what bots may fetch; llms.txt suggests what is worth reading. One is access policy, the other is a curated guide - they complement, never substitute.

What goes in llms-full.txt?

An expanded version carrying actual page content in markdown for systems that want depth in one fetch. Generate it from your real pages at build time so it cannot drift.