On Page SEO - Crawl Rate
A Rough Guide To: LLMS.txt – What It Is & How To Create One
TL;DR: llms.txt is a proposed Markdown file that gives LLMs a simplified overview of the important content on a website. It isn’t required by Google, isn’t a ranking factor, and there’s no guarantee AI platforms will use it. But it’s easy to create, and can provide a clear, machine-readable summary of your key pages and topics. In this guide I’ll show you what goes into one, examples of different structures, and how I’ve built mine.
If you want to create one for your own site, you can use my free llms.txt file generator
What Is LLMS.txt?
llms.txt is a proposed Markdown file designed to give Large Language Models a simplified, structured overview of the important content on a website.
An llms.txt file is commonly placed at the root of a site:
https://example.com/llms.txt
The current proposal also allows llms.txt files within subpaths, such as /docs/llms.txt, where the file describes content beneath that path.
For example, mine is:
https://daveashworth.co/llms.txt
Rather than trying to describe every URL on a website, the file can highlight things such as:
- what the website is about
- important pages
- key topic areas
- supporting resources
- relationships between sections of the site
The idea is to provide LLMs with a cleaner, more concise representation of the website and its important content.
It’s useful to think of llms.txt as a reference document rather than an instruction file. It doesn’t tell an AI crawler what it can or can’t access in the way robots.txt does, and it isn’t a replacement for an XML sitemap.
Most importantly, having an llms.txt file doesn’t mean an AI platform will read or use it. That’s dependent on individual platforms and how they retrieve information.
Does Google Use LLMS.txt?
Google has been fairly clear about its position on llms.txt.
In its guidance on AI features and your website, Google says you don’t need to create new AI-specific files or markup to appear in its AI search features.
Google specifically includes llms.txt in its guidance:
“LLMS.txt files and other ‘special’ markup: You don’t need to create new machine readable files…”
So, at the moment, you don’t need an llms.txt file for Google Search or Google’s AI features. Google has explicitly stated that having one won’t positively or negatively affect your visibility or rankings.
That’s an important distinction because llms.txt is sometimes discussed as though it’s the AI equivalent of robots.txt or an XML sitemap. It isn’t.
An XML sitemap is an established mechanism supported by search engines. llms.txt is a proposed format whose usefulness depends on whether individual AI systems actually choose to use it.
That doesn’t necessarily make llms.txt pointless. Google is only one part of the wider AI ecosystem, and the file can still serve as a concise, structured summary of your website.
Is LLMS.txt Useful?
Potentially, yes, but I think expectations need to stay realistic.
There is currently no guarantee that an AI platform will discover, read or make use of your llms.txt file. Creating one doesn’t mean you’ll suddenly gain citations in ChatGPT, appear more often in AI-generated answers or improve your search rankings.
For me, its usefulness is much simpler.
A well-structured llms.txt file can provide a concise summary of:
- what your website is about
- your main topic areas
- your most important pages
- useful supporting content
- how those areas relate to one another
That potentially gives systems capable of using the file a cleaner representation of your website than having to infer everything from navigation, internal links and individual pages.
There’s also a useful side effect: creating the file forces you to think about your own website structure.
If it’s difficult to decide which pages belong under a particular topic, which pages are important, or how your content relates together, that can expose weaknesses in the site’s organisation itself.
That’s one reason my own llms.txt developed alongside the work I’ve been doing on SEO content hubs. The two serve different purposes, but both require you to think clearly about topics, supporting content and relationships between pages.
So I wouldn’t create llms.txt because I expect an immediate SEO or AI visibility benefit.
I’d create one because it’s easy to implement, provides a clean summary of the website, and may be useful to systems that choose to support it.
How To Create an LLMS.txt File
You don’t need any special software or plugin to create an llms.txt file. At its simplest, it’s a Markdown-formatted text file placed at the root of your website.
You can create one manually using the structure below, or use my free llms.txt file generator to find your sitemap URLs, choose the pages you want to include and generate the file for you.
The proposed format is deliberately straightforward.
1. Add the Website Name and Description
Start with an H1 containing the website or project name, followed by a short description explaining what the site is about.
For example:
# Example Website
> SEO consultancy specialising in ecommerce, technical SEO and analytics.
2. Group Important Pages by Topic
Use H2 headings to organise the main areas of the website, then list the most useful URLs beneath them.
## Core Pages
- [Services](https://example.com/services/): Overview of consultancy services.
- [About](https://example.com/about/): Background and experience.
## Ecommerce SEO
- [Ecommerce SEO](https://example.com/ecommerce-seo/): Ecommerce SEO consultancy and expertise.
- [Magento SEO](https://example.com/magento-seo/): Magento-specific SEO services and guidance.
The descriptions are useful because they provide additional context about what each page contains and why it’s relevant.
3. Include Important Supporting Content
You can also include useful guides, case studies and supporting articles beneath the relevant topic.
For example:
## Supporting Guides
- [Canonicalisation Guide](https://example.com/canonicalisation/): How canonical URLs work and common implementation problems.
- [Crawlability Guide](https://example.com/crawlability/): Identifying and fixing website crawling issues.
You don’t necessarily need to include every URL on the website.
I’d prioritise the pages that best explain your main topics, expertise and useful supporting content, rather than trying to turn llms.txt into another complete sitemap.
4. Upload It to the Root of Your Website
Save the file as:
llms.txt
and make it available at:
https://example.com/llms.txt
It should be publicly accessible without requiring authentication.
That’s really all that’s required to create a basic version.
You can keep it relatively small or build a much more detailed file containing multiple topic areas and supporting resources. The right level of detail depends largely on the size and complexity of the website.
LLMS.txt Example
A basic llms.txt file doesn’t need to be complicated. For a small website, you could simply describe the site and organise the most important pages into a few logical sections.
For example:
# Example Website
> SEO, analytics and ecommerce consultancy. Use the links below to understand the main services, areas of expertise and supporting resources.
## Core Pages
- [Services](https://example.com/services/): Overview of the consultancy services available.
- [About](https://example.com/about/): Background, experience and expertise.
- [Blog](https://example.com/blog/): Guides, case studies and technical resources.
## SEO
- [SEO Consultancy](https://example.com/seo/): SEO services and areas of expertise.
- [Technical SEO](https://example.com/technical-seo/): Technical SEO consultancy and guidance.
## Analytics
- [Analytics](https://example.com/analytics/): Analytics, measurement and tracking services.
- [Google Analytics 4](https://example.com/ga4/): GA4 implementation and optimisation.
## Supporting Guides
- [SEO Guide](https://example.com/seo-guide/): Practical guidance covering common SEO problems.
- [GA4 Guide](https://example.com/ga4-guide/): Guidance on GA4 configuration and tracking.
The important thing isn’t to copy this structure exactly. Your llms.txt should reflect how your own website and its topics are organised.
A small site might only need a handful of sections and URLs. A larger site could group pages into services, products, topic areas, documentation and supporting resources.
I’d avoid trying to list every page simply because it exists. An XML sitemap already serves that purpose. llms.txt is more useful as a curated overview of the content you consider important.
LLMS.txt Best Practices
Because llms.txt is still a relatively new proposal, I wouldn’t over-engineer it. The aim should be to create a file that’s clear, useful and easy to maintain.
A few principles I follow:
Keep the Structure Simple
Use descriptive headings to group related pages rather than creating a complicated hierarchy.
Someone looking at the file should be able to understand the main areas of the website without having to decipher how you’ve organised it.
Prioritise Important Pages
You don’t need to include every URL on the website.
Focus on pages that help explain:
- your main topics
- products or services
- areas of expertise
- useful supporting content
- important documentation or resources
Think curated overview rather than alternative XML sitemap.
Add Useful Descriptions
Don’t just provide a list of URLs. A short description can give additional context about what each page covers.
For example:
- [Magento SEO](https://example.com/magento-seo/): Magento SEO consultancy covering crawlability, indexing, product optimisation and technical SEO.
That’s more informative than simply listing the page title and URL.
Organise Supporting Content by Topic
If you have lots of articles or resources, group them beneath the topics they actually support.
This is particularly relevant if you’ve already organised your website around content hubs. Your llms.txt can reflect those same relationships rather than presenting one enormous list of blog posts.
Don’t Keyword-Stuff It
I wouldn’t treat llms.txt as another place to repeat target keywords.
Write descriptions that accurately explain what pages contain. If the relevant terminology occurs naturally, that’s enough.
Keep It Updated
If you add an important service, remove a page, change a URL or substantially reorganise a topic area, update the file.
A concise file that accurately reflects the current website is more useful than an elaborate one that gradually becomes outdated.
Don’t Expect It to Fix the Website
Most importantly, llms.txt shouldn’t compensate for poor site structure, weak internal linking or unclear content.
It should reflect a well-organised website, not attempt to manufacture one inside a text file.
My Approach to LLMS.txt
My own llms.txt is more detailed than the basic example above because it reflects the broader structure I’ve been building across the website.
Rather than simply listing a selection of URLs, I’ve organised it around areas such as:
- core services
- individual skills
- technical expertise
- supporting blog content
- related topic areas
This developed naturally alongside the work I’ve been doing on SEO content hubs.
For example, if I have a broader Ecommerce SEO topic, I can reference the main Ecommerce SEO page alongside more specific Magento expertise and supporting articles covering related technical problems.
The same principle applies to Local SEO, website performance, analytics and other areas of the site.
The result is effectively a curated map of the topics and expertise across my website, rather than a flat list of URLs.
You can see the complete file here:
I’m not doing this because I expect the file itself to improve rankings or guarantee that my content appears in AI-generated answers.
I’ve created it because the structure already exists across the website, and representing those relationships in a lightweight, machine-readable file is relatively straightforward.
If AI systems make greater use of llms.txt in future, the file is there. If they don’t, I haven’t built my wider SEO or content strategy around something they may never support.
Should You Create an LLMS.txt File?
I think so, provided you keep your expectations realistic.
An llms.txt file is relatively easy to create and maintain, and it gives you another way to provide a clear, structured overview of your website and its most important content.
What I wouldn’t expect it to do is:
- improve rankings simply because it exists
- guarantee visibility in AI-generated answers
- generate citations from ChatGPT or other LLMs
- compensate for weak content or poor site structure
Google has already made it clear that llms.txt isn’t required for its AI search features, and support from other platforms remains uncertain.
So I see it as a low-effort supporting layer, not something to build an AI or SEO strategy around.
If your website already has strong content, clear topic relationships and sensible internal linking, creating an llms.txt file is straightforward enough that I think it’s worth doing.
If broader support develops in future, you’re already prepared. If it doesn’t, you’ve spent relatively little time creating a useful, structured summary of your website.
categories
- Analytics Insights
- Crawl Rate Insights
- Ecommerce SEO Insights
- International SEO Insights
- Keyword Research Insights
- Local SEO Insights
- Migrations Insights
- News Insights
- On Page SEO Insights
- Performance Analysis Insights
- SEO Audits Insights
- Site Speed Insights
- Structured Data Insights
- Tag Manager Insights
- Wordpress SEO Insights
latest posts
How To: Track GA4 AI Assistant Traffic, Including Historical Data
How To: Build SEO Content Hubs for Topical Authority
Case Study: Do Keywords in URLs Help SEO Rankings?
How To: Track Google Business Profile Traffic in GA4
How To: Track Internal Link Clicks in GA4 Using Google Tag Manager
get in touch
If you need some expert website optimisation and configuration, want to find out more, or even just have a question, fill in the form below as I will always be happy to help