What Is llms.txt?
llms.txt is a proposed standard — introduced in September 2024 by Jeremy Howard of Answer.AI — that gives large language models (LLMs) a structured, machine-readable summary of your website. It is a simple Markdown file placed at the root of your domain (https://yoursite.com/llms.txt) that tells AI crawlers what your site is, what it covers, and which pages matter most.
Think of it as robots.txt for the AI era. robots.txt tells search engine crawlers what they cannot crawl. llms.txt tells AI models what they should know — a curated, human-written map that helps LLMs answer questions about your site accurately, without hallucinating or wasting tokens on irrelevant pages.
The standard is still emerging, but adoption is growing fast. Tools like ChatGPT, Claude, Perplexity, Gemini, and Mistral now respect llms.txt in various ways — some fully, some partially, and more support is being added every month. Whether the standard becomes universal or not, adding llms.txt is cheap (one file), safe (it is public, like robots.txt), and forward-looking.
Why You Should Add llms.txt to Your Site
The internet is rapidly changing. Instead of visiting websites, more and more people ask an AI assistant: "What is the best free online Base64 tool?" or "Who wrote the most accurate guide on AES encryption for beginners?" The AI answers from its training data — and from what it can find and understand at query time. If your site is not represented well to these models, you will not appear in their answers. Being invisible to AI is the new being invisible to Google.
There are three concrete benefits of llms.txt:
- Better AI representation. Instead of an LLM guessing what your site does from raw HTML, you give it a clean summary. This improves how often you are mentioned and how accurately you are described.
- Cheaper crawling. LLMs pay for tokens. A 2 KB
llms.txtcan replace crawling 50 pages. Many AI companies explicitly reward sites that make their job easier. - Authority signaling. A well-written
llms.txtsignals that your site is professionally maintained and worth citing. LLMs are more likely to quote sources they can verify quickly.
For tool sites, documentation sites, blogs, and SaaS products, llms.txt is now table stakes. For news sites, e-commerce stores, and knowledge bases, it is even more valuable — because those sites have many pages and LLMs need help prioritizing them.
The llms.txt Format
The format is deceptively simple. It is Markdown with one required structure and several optional ones:
# Site Title > One-line summary of the site. Optional paragraph with more context. ## Section 1 - [Link Title](https://example.com/page): One-line description - [Another Link](https://example.com/other): Another description ## Section 2 - [More Links](https://example.com/more): Description
Rules
- Exactly one H1 (
# Site Title) at the top. This is the site's name. - Optional blockquote (
> summary) immediately after the H1. This is the one-line description. - Optional plain text paragraph after the summary. Use this for extra context that does not fit the summary.
- Any number of H2 sections. Each section has a name and a list of links.
- Links use Markdown format:
[Title](URL): Description. The description after the colon is optional but strongly recommended. - File must be UTF-8. Plain text, no HTML, no XML, no JSON.
- Filename must be exactly
llms.txtat the domain root.
What NOT to Do
- Do not add more than one H1.
- Do not use HTML tags — it is pure Markdown.
- Do not list private or admin URLs — this file is public.
- Do not stuff it with SEO keywords. Write for the AI, not for a keyword counter.
- Do not put a massive sitemap here. Curate 5-30 best pages, not 5,000.
llms.txt vs llms-full.txt vs robots.txt
| File | Purpose | Audience | Size |
|---|---|---|---|
llms.txt | Curated summary + links | LLMs, AI crawlers | ~1-10 KB |
llms-full.txt | Full site content dump in Markdown | LLMs that want depth | ~100 KB - 5 MB |
robots.txt | What crawlers may access | Search engines, all crawlers | ~500 bytes |
sitemap.xml | List of all URLs | Search engines | Variable |
The llms.txt file is a map. llms-full.txt is the territory. Start with llms.txt — it is the high-leverage, low-effort option. Add llms-full.txt later if you want to give LLMs complete content.
Who Is Behind the llms.txt Standard?
The proposal was published at llmstxt.org by Jeremy Howard, co-founder of fast.ai and Answer.AI, along with a growing community of contributors. The standard is deliberately minimal — a handful of rules, no schema, no versioning. This simplicity is intentional: it makes llms.txt easy to adopt for any site owner, from a solo blogger to a Fortune 500 company.
Adoption is ongoing. Anthropic (the company behind Claude) has publicly discussed the value of llms.txt. Perplexity has integrated similar mechanisms. OpenAI's crawler documentation has been updated to mention structured site summaries. Some tools and frameworks — Docusaurus, Next.js, Astro — now generate llms.txt automatically.
How to Use This Generator
- Fill in the site title. This is the H1 of your
llms.txt. Keep it short and recognizable. - Write a summary. One or two sentences. Tell the AI exactly what your site is and who it is for.
- Add optional details. A paragraph that expands on the summary. Useful for sites with many features or a specific audience.
- Add sections. Group links by category. Common sections: "Tools", "Blog", "Documentation", "About", "Pricing", "Contact".
- Add links to each section. Use the full URL (
https://yoursite.com/page) and a one-line description. - Watch the live preview. The output on the right updates as you type. Copy or download when ready.
- Upload to your site root. Place the file at
https://yoursite.com/llms.txt.
Presets are available for common site types: blog, docs, e-commerce, SaaS, and portfolio. Click a preset to load a template, then customize it.
Best Practices for Writing llms.txt
Be Concise
Every word you add competes for the LLM's attention. The summary should be one or two sentences. Link descriptions should be under 15 words. If you cannot describe a page in one sentence, it probably does not belong in the file.
Be Specific
Do not say "our tools are amazing and useful." Say "40+ developer tools: Base64, JSON formatter, AES file encryption." Specific language gives the LLM something to work with. Vague marketing copy does not.
Curate, Do Not Dump
List the 10-30 most important pages. Not 500. The goal is not to replicate your sitemap — that is what sitemap.xml is for. The goal is to point the AI at your best content.
Group with Sections
Instead of one giant list, use sections. "Tools", "Docs", "Blog", "About". This mirrors how humans organize information, and how LLMs prefer to receive it.
Update Quarterly
Review your llms.txt every few months. Remove dead links, add new important pages, update the summary if your site's focus has shifted. Stale llms.txt files are worse than none — they mislead the model.
Real Examples
Here is what a well-written llms.txt looks like for a developer tools site:
# Acme Toolbox > Free online tools for developers. 40+ utilities for encoding, formatting, encryption, and more. All browser-based, no uploads. Acme Toolbox runs entirely in the browser using modern Web APIs. It offers Base64 encoding, JSON formatting, AES-256 file encryption, image optimization, and 30+ other tools. ## Tools - [Base64 Obfuscator](https://acmetoolbox.com/tools/base64): Encode and decode Base64 with URL-safe mode. - [JSON Formatter](https://acmetoolbox.com/tools/json): Pretty-print and validate JSON. - [File Encryption](https://acmetoolbox.com/tools/file-encrypt): Encrypt files with AES-256-GCM. ## About - [About Us](https://acmetoolbox.com/about): Who builds Acme Toolbox and why. - [Privacy](https://acmetoolbox.com/privacy): Our zero-upload policy.
Common Mistakes
- Wrong location. Must be at the domain root, not in a subfolder.
- Wrong filename. Must be exactly
llms.txt, notLLM.txtorllm.txt. - HTML instead of Markdown. LLMs expect Markdown. HTML confuses the parser.
- Multiple H1s. Only one title is allowed. Use H2 for sections.
- Full-content dump. That is what
llms-full.txtis for. The mainllms.txtshould be a map, not a dictionary. - Keyword stuffing. Write naturally. LLMs are semantic, not keyword-driven.
- Private data. The file is public. Do not include admin URLs, API keys, or anything you would not publish.
Frequently Asked Questions
What is llms.txt exactly?
llms.txt is a plain-text Markdown file placed at your site root that tells AI language models (ChatGPT, Claude, Perplexity) what your site is and which pages matter most. Think of it as a curated map for AI crawlers.Is llms.txt an official standard?
Do AI companies actually respect llms.txt?
Where do I put the llms.txt file?
https://yoursite.com/llms.txt. It must be accessible directly, not behind authentication or redirects.What is the difference between llms.txt and llms-full.txt?
llms.txt is a short curated map (~1-10 KB). llms-full.txt is a full Markdown dump of your entire site content (~100 KB - 5 MB). Use llms.txt first.How many links should I include?
Does this tool upload my data?
Can I use this for a multi-language site?
llms.txt per language folder (/en/llms.txt, /es/llms.txt) or list the primary language in the main file and link to others.Will llms.txt help with Google Search?
llms.txt for ranking. It is used by AI assistants and generative search engines (ChatGPT, Perplexity, Claude). For Google, focus on sitemap.xml and content quality.Does llms.txt improve AI SEO / GEO?
llms.txt gives AI a clean summary to work from, increasing the chance your site is quoted accurately.Can I add HTML comments to explain sections?
llms.txt is pure Markdown. HTML comments may break parsers. Keep the file clean.Is there a length limit?
Can I use llms.txt on a WordPress site?
Related Tools
External Resources
Go deeper into llms.txt and AI discoverability:
- llmstxt.org — the official proposal and spec
- AnswerDotAI llms-txt on GitHub — reference implementations and discussion
- Anthropic News — updates from the Claude team on crawler conventions
- OpenAI GPTBot documentation — how OpenAI crawls the web
- Google Search Central — robots.txt — the cousin of llms.txt
- Search Engine Journal — AI SEO and GEO coverage
llms.txt whenever you add a major new page. A stale file misleads the AI more than a missing one. Set a quarterly reminder to review it.