What llms.txt is and whether you need one

If you've read about AI search recently, you may have seen advice to add an "llms.txt" file to your website, sometimes presented as essential. This guide explains what it actually is, where it came from, and whether it's worth your time.

What it is

llms.txt was proposed in 2024 by Jeremy Howard of Answer.AI. The idea: websites are built for human browsers, full of navigation, scripts and layout that make them awkward for language models to read. A simple text file could point AI tools straight to the most useful content.

The proposal describes a file at yoursite.com/llms.txt, written in Markdown, containing:

The site or project name as a heading.
A short summary.
Optional extra detail.
Lists of links to important pages, ideally to clean Markdown versions of those pages.

It's designed mainly with documentation-heavy sites in mind, such as software libraries, where a developer might want to load a project's docs into an AI coding assistant.

What it isn't

It isn't a standard. It hasn't been adopted by a standards body. It's a proposal published at llmstxt.org.
It isn't like robots.txt. robots.txt is a long-established convention, formalized as an internet standard (RFC 9309), that crawlers check for permission rules. llms.txt doesn't grant or block access to anything.
It isn't a confirmed ranking or citation signal. None of the major AI search providers has documented that their search or answer features read it.

Who uses it

Some companies, particularly developer tools and documentation sites, publish an llms.txt file, and some documentation platforms can generate one automatically. Some AI coding tools and individual developers use these files when they deliberately want to load a project's documentation.

That's different from AI search engines reading it on their own when deciding what to cite. On that, the public picture as of this writing:

Google documents what's needed for its AI features (being indexed and eligible for a snippet) and says there are no additional requirements. It doesn't list llms.txt, or any other special file, as something you need.
OpenAI, Perplexity and others document their crawlers and robots.txt controls. Their published crawler documentation doesn't say they use llms.txt for search.

Things in this area change, so if a provider announces support, that changes the calculation. Check the providers' own documentation rather than relying on blog posts.

Should you add one?

Help me choose

Should you add an llms.txt file?

Do you publish extensive technical documentation, an API or a software product developers use?

You probably don't need one if you're a typical local or service business. Your time is better spent on the things AI tools demonstrably rely on: being indexed, clear content, accurate business information and a good reputation. See Is SEO dead?.

It might be worth it if you publish extensive technical documentation, an API, or a software product that developers will want to load into AI assistants. In that case an llms.txt that points to clean versions of your docs can be genuinely useful to those users.

If you add one anyway:

Keep it short and accurate.
Link only to public pages you're happy for anyone to read.
Don't put anything in it you wouldn't put on your website. It's public.
Keep it updated, or it becomes another place for outdated information to live.

What to do instead (or as well)

Check robots.txt. Make sure you're not blocking the AI search crawlers you want to be found by, such as OAI-SearchBot and PerplexityBot, or Googlebot and Bingbot. See Sitemaps & robots.txt explained.

Keep your XML sitemap healthy and submitted to Google Search Console and Bing Webmaster Tools.

Write clear, quotable content. See Writing content AI assistants can quote.

Keep your business facts consistent across your site, structured data, Google Business Profile and directories. See Structured data explained.

Frequently asked questions

Someone is offering to "optimize my site for AI" with an llms.txt file. Is that worth it?

Ask them which AI search provider has documented using it, and to show the documentation. Adding the file is quick work. A service that charges significantly for it, or presents it as the core of AI visibility, is overselling.

Can llms.txt stop AI companies using my content?

No. It's not a permissions file. To control crawler access, use robots.txt rules for the specific bots, as documented by each company. Some providers also offer separate controls for training versus search.