llms.txt Explained: What It Is, How to Ship It, Why It Matters

llms.txt is a markdown file at the root of your domain, https://yourdomain.com/llms.txt; that lists your most important URLs with one-line descriptions, designed for AI engines and LLM agents that want a curated index of your site without crawling everything. The companion llms-full.txt concatenates the full markdown content of those URLs into a single file.

I am Ignacio (IGNAX), Spain-born and Paraguay-based, a solo full-stack developer and SEO/AEO specialist. Both files on this site are generated automatically from the content registry on every build. For the conceptual framing of AEO see what is AEO.

What is llms.txt, in one paragraph?

llms.txt is a proposed open standard introduced at llmstxt.org by Jeremy Howard. The format is intentionally simple: a markdown file with a top-level H1 (site name), an optional blockquote intro, then sections of links with one-line descriptions per URL. It functions as a curated table of contents for AI agents and LLM crawlers; the equivalent of sitemap.xml for AI rather than for search engines.

Why does llms.txt exist?

Using sitemap.xml as an AI crawling surface runs into three problems:

• Sitemap.xml is undifferentiated. Every URL gets the same weight. Your privacy policy and your flagship article look identical to a crawler.
• No descriptions. XML sitemaps have no place for a human-readable explanation of what the page is.
• No content. Sitemap.xml lists URLs; an LLM that wants the actual content has to fetch each URL separately.

llms.txt addresses all three. The file is curated (only the URLs you want quoted), descriptive (one-line description per link, section grouping), and the companion llms-full.txt contains the actual content for one-shot ingestion.

What does a llms.txt file look like?

Here is the working file at https://ignax.dev/llms.txt:

# Ignax.dev

> Solo full-stack developer and SEO/AEO specialist. SaaS MVPs, RAG chatbots, multilingual SEO. Spain-born, Paraguay-based, bilingual EN/ES.

## Services

- [SaaS MVP Development](https://ignax.dev/services/saas-mvp): Production SaaS MVPs in 2–6 weeks for $3,000–$15,000 USD
- [SEO + AEO Setup](https://ignax.dev/services/seo-aeo-setup): Get cited by ChatGPT, Perplexity, Gemini, Claude
- [RAG Chatbots](https://ignax.dev/services/rag-chatbots): Production retrieval-augmented chatbots on Postgres + pgvector
- [Multilingual SEO](https://ignax.dev/services/multilingual-seo): Bilingual EN/ES SEO with hreflang done right
- [Technical SEO Audit](https://ignax.dev/services/technical-seo-audit): Crawl, schema, Core Web Vitals, prioritized fix list

## Articles

- [What is AEO?](https://ignax.dev/articles/what-is-aeo): Answer Engine Optimization explained
- [How to Get Cited by ChatGPT](https://ignax.dev/articles/how-to-get-cited-by-chatgpt): The concrete checklist
- [Build a SaaS MVP Fast](https://ignax.dev/articles/build-saas-mvp-fast): The exact sequence I run on paid builds
- [RAG vs Fine-Tuning for Chatbots](https://ignax.dev/articles/rag-vs-fine-tuning-chatbots): When each architecture wins

## Case Studies

- [Growth Engine](https://ignax.dev/case-studies/ignax-dev-growth-engine): The live AEO experiment running on this site
- [AI Lead Generation Platform](https://ignax.dev/case-studies/ai-lead-generation-platform): SaaS built in 4 weeks

That is the entire spec. No XML, no namespaces, no schemas. Markdown, links, descriptions.

What does llms-full.txt look like?

llms-full.txt is the same set of URLs but with the full markdown content concatenated. The format I emit:

# Ignax.dev, Full Content Bundle

---

## /services/saas-mvp

<full markdown of the SaaS MVP service page>

---

## /services/seo-aeo-setup

<full markdown of the AEO setup service page>

---

## /articles/what-is-aeo

<full markdown of the AEO explainer article>

Each section is separated by ---, prefixed with the URL path as an H2. An LLM agent can fetch this single file and ingest the whole site in one HTTP request. For a 50-page site, llms-full.txt lands at roughly 200 to 500 KB, easily within the context window of any modern frontier model.

How do I generate llms.txt automatically?

Hand-editing is a trap. Generate from your content registry on every build. Here is the SvelteKit endpoint I use:

// src/routes/llms.txt/+server.ts
import { loadAllContent } from '$lib/content/loader';
import { SITE } from '$lib/seo/site';

export const prerender = true;

export async function GET() {
  const all = await loadAllContent();
  const services = all.filter((c) => c.type === 'service' && c.lang === 'en');
  const articles = all.filter((c) => c.type === 'article' && c.lang === 'en');
  const cases = all.filter((c) => c.type === 'case-study' && c.lang === 'en');

  const lines: string[] = [];
  lines.push(`# ${SITE.name}`);
  lines.push('');
  lines.push(`> ${SITE.description}`);
  lines.push('');

  if (services.length) {
    lines.push('## Services');
    lines.push('');
    for (const s of services) {
      lines.push(`- [${s.title}](${SITE.url}/services/${s.slug}): ${s.description}`);
    }
    lines.push('');
  }

  // ... Articles, case-studies, etc.

  return new Response(lines.join('\n'), {
    headers: { 'content-type': 'text/plain; charset=utf-8' },
  });
}

The principle: one source of truth (the content registry), many derived outputs (sitemap.xml, llms.txt, llms-full.txt, individual pages). Drift becomes impossible.

For the llms-full.txt variant, swap the per-URL line for the full markdown body of each entry, same loop, different rendering. See the SEO + AEO setup service for the full pattern.

Where should llms.txt live?

At the root of your domain: https://yourdomain.com/llms.txt. Not in a subdirectory, not under a path, not behind a redirect. The proposed spec is unambiguous on this; AI engines look at the root path.

Make sure:

• HTTP 200, no redirects.
• Content-Type: text/plain; charset=utf-8 (or text/markdown).
• No auth wall, no Cloudflare challenge page.
• File is publicly reachable from a curl command without headers.

Validate by visiting the URL in a fresh browser session and running curl -I https://yourdomain.com/llms.txt from a terminal. If the response is anything but 200 OK with plain text, fix it.

Should I include llms.txt in robots.txt?

The convention is still settling. Some aggregators look for a Llms-Sitemap: directive analogous to Sitemap:. Others discover llms.txt by convention. I include both, costs nothing:

User-agent: *
Allow: /

Sitemap: https://ignax.dev/sitemap.xml

For the broader robots.txt + AEO discussion see how to get cited by ChatGPT.

What should I include in llms.txt?

Curate. The signal is in what you exclude.

Include:

• Services, products, pricing pages.
• Long-form articles and tutorials (the AEO-targeted ones especially).
• Case studies with concrete results.
• About page with author entity.
• Documentation entry points.

Exclude:

• Tag archives and category pages.
• Paginated lists.
• Search results pages.
• Privacy policy, terms of service (unless legally relevant to your business).
• Author archive pages.
• Any page that exists for navigation rather than content.

A 30 to 50 URL llms.txt of curated high-value pages produces better citation outcomes than a 500-URL dump. The model uses the list to decide what to read; junk in the list dilutes the trust.

Does llms.txt actually get used by AI engines?

The honest read in 2026.

• Anthropic has publicly acknowledged llms.txt and ClaudeBot appears to respect it.
• OpenAI has not formally endorsed it but GPTBot has been observed fetching /llms.txt in server logs across multiple deployments.
• Google has not endorsed it; Google-Extended crawler behavior is less observable.
• Perplexity has been observed fetching /llms.txt in logs.

The cost of shipping is essentially zero when you auto-generate. The expected value is positive even if only one engine uses it. Skip it only if you also skip your sitemap, which nobody does.

What external references should I read?

• llmstxt.org; the proposed standard, with examples and rationale.
• Anthropic documentation on web crawling: ClaudeBot specifics.
• OpenAI GPTBot documentation: GPTBot crawler behavior.
• Schema.org WebSite type; the schema-side counterpart that complements llms.txt.

Ship llms.txt and llms-full.txt as build-time generated endpoints. Curate the URL list. Validate the file at the root. Move on. The next AEO step is how to get cited by ChatGPT and the conceptual framing is what is AEO.

Ready to wire llms.txt into your stack? Email hello@ignax.dev with your site URL. I generate the file the same day on most stacks.