llms.txt: the standard file that helps AI understand your website
llms.txt is an open specification for providing LLM-friendly markdown content at /llms.txt. Learn how this simple standard helps AI assistants like ChatGPT, Claude, and Gemini understand your site better at inference time.
AI standardsllms.txtLLM optimizationDocumentationDeveloper tools
Large language models are changing how people find information online. But there is a fundamental problem: LLMs can't read your entire website. Context windows are limited, HTML is messy, and converting pages with navigation, ads, and JavaScript into clean text is difficult.
llms.txt is an open specification designed to solve this. It is a simple markdown file at your site's root (/llms.txt) that provides LLM-friendly content: brief summaries, structured guidance, and links to detailed markdown files.
Think of it as a curated introduction to your site, specifically for AI assistants.
Why llms.txt exists
When an AI assistant like ChatGPT or Claude tries to understand your website, it faces several challenges:
Context window limits - Most LLMs can't process entire websites at once
HTML complexity - Navigation, ads, JavaScript, and styling add noise
No clear starting point - Which pages matter most for understanding your project?
Imprecise conversion - Turning HTML to plain text loses structure and context
llms.txt addresses these by providing:
A single, accessible location for LLM-optimized content
Clean markdown that both humans and models can read easily
Structured metadata that classical parsers can handle (not just LLM inference)
Links to detailed documentation in markdown format
The llms.txt specification
The format is deliberately simple. Here is what goes in /llms.txt:
Required sections
H1 heading - Your project or site name
Blockquote summary - Brief overview containing key information
Optional sections
Additional details - Paragraphs, lists, or other markdown (no headings)
File lists - H2-delimited sections with markdown lists of URLs
There is also an "Optional" section (H2: ## Optional) for secondary content that LLMs can skip when they need shorter context.
Example structure
markdown
# FastHTML> FastHTML is a Python library combining Starlette, Uvicorn, HTMX, and fastcore's FT into a framework for server-rendered hypermedia applications.
Important notes:
- Not compatible with FastAPI syntax
- Works with vanilla JS and web components, not React/Vue/Svelte
## Docs- [FastHTML quick start](https://docs.example.com/quickstart.html.md): Brief overview
- [HTMX reference](https://github.com/bigskysoftware/htmx/blob/master/www/content/reference.md): All HTMX attributes and methods
## Examples- [Todo app walkthrough](https://github.com/example/app.py): Complete CRUD app showing FastHTML patterns
## Optional- [Starlette documentation](https://example.com/starlette.md): Subset useful for FastHTML development
Markdown versions of pages (.md convention)
The specification also recommends providing markdown versions of your HTML pages by appending .md to URLs:
This gives LLMs access to clean, text-based versions without HTML overhead.
Who should use llms.txt
This specification is particularly valuable for:
Software documentation sites
If you maintain API docs, SDK guides, or technical tutorials, llms.txt helps AI assistants answer developer questions correctly. Example: "How do I authenticate with the API?"
Open-source projects
GitHub repositories with extensive READMEs, wikis, or doc sites benefit from structured llms.txt files that guide LLMs to installation guides, usage examples, and API references.
Educational content
Tutorial sites, course platforms, and knowledge bases can use llms.txt to surface key learning paths and reference materials.
E-commerce sites
Product catalogs, return policies, and FAQ pages can be surfaced through llms.txt so AI assistants give accurate information about your offerings.
Business websites
Company structure, services offered, team info, and case studies can be organized so LLMs understand your business accurately.
Generate expanded context files and test whether models can:
Answer basic questions about your project
Find specific documentation
Understand your API structure
Explain core concepts accurately
5. Keep it updated
llms.txt should evolve with your project. When you add major features or documentation sections, update the file accordingly.
Expanded context files (llms-ctx.txt)
Some projects generate expanded versions that concatenate the linked content:
llms-ctx.txt - Expanded content without "Optional" section URLs
llms-ctx-full.txt - Expanded content including "Optional" URLs
These files are created by tools like llms_txt2ctx that parse the llms.txt file and fetch linked content. This is useful for:
AI agents that need full context upfront
Testing how well your documentation answers questions
Providing precomputed context for specific use cases
The FastHTML project uses this approach, generating XML-structured context files from their llms.txt.
Current adoption and ecosystem
Adoption status
As of 2026, llms.txt adoption is in early stages:
No official support from major AI providers (OpenAI, Anthropic, Google) has been announced
Limited crawling - Server logs show minimal bot traffic specifically requesting llms.txt
Growing awareness - Developer communities are discussing and implementing it
Proactive projects are adding it as future-proofing
Community tools and integrations
Several tools support llms.txt:
llms_txt2ctx - CLI and Python module for parsing and expanding llms.txt files
JavaScript implementation - Sample JS parser for the spec
VitePress plugin - Auto-generates llms.txt for VitePress documentation sites
Docusaurus plugin - Generates LLM-friendly docs following the spec
Drupal Recipe - Full llms.txt support for Drupal 10.3+ sites
llms-txt-php - PHP library for reading and writing llms.txt files
VS Code PagePilot Extension - Chat participant that loads llms.txt context automatically
Directories
Two directories catalog llms.txt files on the web:
llmstxt.site
directory.llmstxt.cloud
Criticism and trade-offs
The SEOPub critique
Some SEO professionals have raised concerns:
"They don't benefit your website at all. They only benefit the LLM providers. There is no reference to original URLs, so if an LLM cites you, the link goes to the .md file - just a wall of text. Horrible user experience."
This is a valid concern. The spec should be extended to:
Ensure .md versions include canonical URL references
Provide guidance on citation formatting
Consider how LLM-generated citations link back to HTML pages, not markdown sources
The John Mueller perspective
Google's John Mueller compared llms.txt to the meta keywords tag:
"This is what a site-owner claims their site is about... At that point, why not just check the site directly?"
Fair point. LLMs could theoretically parse HTML directly. But llms.txt advocates argue:
Context windows make full-site parsing impractical
Clean markdown is easier to process than HTML
Curated content reduces noise and improves accuracy
Semantic structure helps models reason about content
The "too early" argument
Many developers note that it is premature to invest heavily in llms.txt since:
No major AI service officially uses it
The spec might evolve or be replaced
Effort could be better spent on traditional SEO and user-facing docs
Counter-argument: early adopters shape emerging standards. If llms.txt gains traction, sites with existing files will benefit first.
How llms.txt compares to other standards
robots.txt
Purpose: Control automated crawler access
Audience: Search engine bots
Use case: Prevent indexing of specific paths
sitemap.xml
Purpose: List all indexable pages
Audience: Search engine crawlers
Use case: Help search engines discover and index content
llms.txt
Purpose: Provide curated, LLM-friendly content
Audience: Large language models (ChatGPT, Claude, Gemini)
Use case: Help AI assistants understand your site during inference
Key difference: llms.txt is for inference time, not training or indexing. It helps when a user asks an AI assistant a question and the model needs to understand your site right now.
Should you add llms.txt to your site?
Yes, if you:
Run a documentation site with frequent AI-related queries
Want to future-proof your site for LLM-based discovery
Have complex content that benefits from curation
Can generate markdown versions of your pages easily
Are experimenting with AI-native workflows
Maybe not, if you:
Have a simple, single-page site
Lack resources to maintain markdown versions
Prefer to wait for official LLM provider support
Focus on traditional search traffic
A balanced approach
Even if llms.txt is not widely adopted today, the practice of creating clean, structured markdown documentation is valuable on its own. You can:
Create a basic llms.txt file (low effort, future-proofing)
Generate markdown versions of key pages (useful for LLMs and human readers)
Monitor logs for llms.txt requests (track if AI services start using it)
Update the file as your docs evolve
Worst case: you have better-organized documentation. Best case: you are ready when LLMs standardize on this approach.
Creating your llms.txt file
Manual approach
Create /llms.txt in your site root
Write H1, blockquote summary, and optional details
Add H2 sections with file lists
Generate .md versions of key pages
Test the file by expanding it and asking LLMs questions
Automated tools
Use plugins or CLI tools:
VitePress plugin for VitePress sites
Docusaurus plugin for Docusaurus sites
Drupal Recipe for Drupal sites
Custom build scripts for static site generators
Generator tool
For a quick start, use the llms.txt generator on explainx.ai - a free tool that creates properly formatted llms.txt files based on your project details.
The future of llms.txt
The specification is community-driven and open for input. A GitHub repository hosts the informal overview, and a Discord channel facilitates discussion.
For llms.txt to succeed, it needs:
LLM provider adoption - OpenAI, Anthropic, Google, and others officially supporting it
Tooling maturity - Better generators, validators, and analytics
Clear value proposition - Measurable impact on LLM response accuracy
Spec evolution - Address citation, attribution, and URL reference issues
Until then, it remains an experiment worth watching - and potentially participating in.
llms.txt is a simple, structured way to help AI assistants understand your website. While adoption is early and not all major LLM providers officially support it yet, the underlying idea is sound: give models clean, curated content so they can provide accurate answers about your project.
Whether llms.txt becomes the standard or not, creating well-organized, markdown-based documentation benefits both human readers and AI systems. If you are already maintaining docs, adding a llms.txt file is a low-effort experiment that could pay off as LLM-based discovery becomes more common.
The spec is open, the tools exist, and the worst-case outcome is slightly better documentation. That is a reasonable bet on the future of AI-assisted information retrieval.
