If you manage SEO, AEO, GEO or digital experience for a Sitecore website, you may have started hearing about a new file:
llms.txt
The idea sounds straightforward.
We already have robots.txt to communicate crawling instructions.
We have sitemap.xml to help search engines discover URLs.
So perhaps we now need an llms.txt file to help ChatGPT, Gemini, Claude, Perplexity and other AI systems understand our websites.
That naturally leads to a question I increasingly hear:
Should I add an llms.txt file to my Sitecore website?
My answer today is:
Yes, you can — and for some organizations it is worth experimenting with. But I would not treat llms.txt as a major AEO ranking factor, nor would I prioritize it ahead of fixing the fundamentals of your website.
There is a lot of nuance behind that answer.
⸻
First: What exactly is llms.txt?
llms.txt is a proposed convention for publishing a concise, machine-readable guide to the most important information on a website.
It normally lives at:
https://www.example.com/llms.txt
The proposal was originally introduced in 2024 and has continued to evolve. The current specification describes llms.txt as a Markdown-formatted file that gives AI agents a high-level description of a website and links them to useful resources. (llms-txt)
A simple file might look something like this:
# Acme Corporation
> Acme manufactures industrial materials for automotive,
> healthcare and consumer applications.
## About Acme
– [About Us](https://www.example.com/about)
– [Leadership](https://www.example.com/leadership)
## Products
– [Product Portfolio](https://www.example.com/products)
– [Material Solutions](https://www.example.com/materials)
## Resources
– [Technical Documentation](https://www.example.com/resources)
– [Research](https://www.example.com/research)
## Sustainability
– [Sustainability](https://www.example.com/sustainability)
– [ESG Report](https://www.example.com/esg)
Think of it less like a traditional XML sitemap containing thousands of URLs and more like a curated orientation guide for an AI agent.
That distinction is important.
⸻
llms.txt is NOT robots.txt for AI
This is probably the biggest misconception.
robots.txt tells crawlers what they may or may not crawl.
For example:
User-agent: *
Disallow: /internal/
llms.txt does not perform that function.
It does not currently provide a standardized permission mechanism telling AI systems:
“You may train on this content.”
or:
“You may use this page in an AI answer.”
Those are separate issues.
Sitecore itself has long supported robots.txt behavior through SXA, including crawler controls and sitemap references. (Sitecore Documentation)
So I would think about the files this way:
robots.txt
Controls crawler access.
sitemap.xml
Helps machines discover your pages.
structured data
Provides explicit information about entities, products, organizations, articles and other content.
llms.txt
Potentially provides AI agents with a curated map explaining which resources on your site are particularly useful.
These mechanisms should complement each other.
They are not interchangeable.
⸻
Does Google use llms.txt?
This is where the conversation becomes particularly important for SEO and AEO teams.
Google has now been very explicit about this.
Google Search does not require llms.txt for inclusion in its AI-powered search experiences.
Google’s current guidance says there are no additional technical requirements for appearing in AI Overviews or AI Mode beyond the fundamentals required for Google Search. Google specifically states that website owners do not need to create new AI-specific machine-readable files to participate. (Google for Developers)
Google went even further in its generative AI optimization guidance.
It specifically addressed llms.txt and said that Google Search does not use the file. Creating one will neither improve nor hurt your Google Search visibility or rankings. (Google for Developers)
That point deserves emphasis:
Adding llms.txt is not going to magically improve your Google AI visibility.
If someone tells you:
“Install an llms.txt file and your brand will start appearing in AI Overviews.”
I would be very skeptical.
That is not how this works.
⸻
Then why are people talking about llms.txt?
Because Google is only one part of the emerging AI discovery ecosystem.
AI agents increasingly need to navigate websites themselves.
Imagine someone asking an AI assistant:
“Compare the sustainability commitments of these three manufacturers.”
Or:
“Find the installation guide for this product.”
Or:
“Which of this company’s materials are suitable for food-contact applications?”
An agent might have to search a website, navigate pages, identify authoritative information and extract the relevant content.
Large enterprise websites can make that surprisingly difficult.
You may have:
- tens of thousands of pages,
- multiple languages,
- multiple brands,
- regional sites,
- PDFs,
- technical documentation,
- investor information,
- product databases,
- campaign landing pages,
- outdated pages,
- duplicate content,
- personalization,
- JavaScript-heavy rendering,
- and several generations of information architecture.
That is extremely common in enterprise Sitecore implementations.
An llms.txt file potentially gives an AI agent a shortcut:
Here are the authoritative areas of this website. Start here.
The current llms.txt specification is explicitly aimed at helping agents locate the most useful information and then follow links to the underlying resources. (llms-txt)
That is where I believe the idea becomes interesting.
⸻
Why Sitecore websites are particularly interesting
Sitecore is often used by large enterprises.
And large enterprises rarely have simple websites.
A typical Sitecore estate might contain:
Corporate content
About the company, leadership, locations and corporate information.
Product content
Hundreds or thousands of product pages.
Brand content
Multiple brands running from the same Sitecore ecosystem.
Regional content
United States, Canada, UK, Germany, Brazil, India and other market variations.
Multilingual content
Different language versions of similar pages.
Technical content
Specifications, data sheets, installation guides and documentation.
News and thought leadership
Articles, press releases and research.
Investor content
Annual reports, earnings and governance information.
When an AI system encounters this environment, simply giving it a sitemap containing 50,000 URLs may not be particularly helpful.
An llms.txt file could theoretically function as a curated semantic entry point into that ecosystem.
That is why I wouldn’t immediately dismiss it.
⸻
But there is an important problem
There is currently a danger that organizations will treat llms.txt as a checkbox.
Someone in SEO hears about it.
A Jira ticket gets created:
“Implement llms.txt for AEO.”
A developer creates the file.
It contains 50 or 100 links.
The ticket gets closed.
Everyone moves on.
That creates very little value.
The real question should be:
What information would we want an AI agent to understand about our organization and where should that agent go to verify it?
That is a very different exercise.
⸻
Don’t put your entire sitemap into llms.txt
If your XML sitemap contains 25,000 pages, copying those 25,000 URLs into llms.txt completely misses the point.
You already have a sitemap.
Your llms.txt should instead be curated.
For example, an enterprise manufacturer might organize it around:
# Company Name
> Company Name is a global manufacturer specializing in…
## Company
Official corporate information and company history.
## Products
Primary product families and product discovery pages.
## Technical Resources
Specifications, technical documentation and safety information.
## Sustainability
Official sustainability commitments and reports.
## Research
Research, scientific publications and technical expertise.
## Newsroom
Official announcements and corporate news.
## Contact
Official customer service and company contact information.
Now the file is actually helping an agent understand the structure of your information ecosystem.
⸻
This becomes a content governance question
This is one reason I believe llms.txt should not be owned exclusively by developers.
A developer can implement the file.
But developers should probably not decide which claims and content represent the authoritative description of the enterprise.
That requires collaboration between:
- SEO/AEO teams,
- content teams,
- product teams,
- corporate communications,
- legal,
- brand teams,
- digital product owners,
- and engineering.
For enterprise organizations, I would actually treat llms.txt as an information-governance artifact.
The technical implementation is easy.
The governance is harder.
⸻
What should go inside your llms.txt?
For a corporate Sitecore website, I would start with approximately six categories.
1. Organization
Link to canonical information about the company.
For example:
- About Us
- Company History
- Leadership
- Locations
- Corporate Facts
The objective is to help machines identify the primary entity behind the website.
⸻
2. Products or services
Identify your major commercial offerings.
Avoid listing every individual SKU unless there is a good reason.
Instead, link to the strongest category or portfolio pages.
For example:
## Products
– [Personal Care](…)
– [Professional Solutions](…)
– [Industrial Materials](…)
– [Healthcare Solutions](…)
This provides a much cleaner representation of the company’s portfolio.
⸻
3. Expertise
This is especially important for AEO.
What topics does the company genuinely have expertise in?
Link to:
- research,
- technical articles,
- expert resources,
- white papers,
- scientific content,
- methodologies,
- FAQs where appropriate.
This is often far more useful than simply linking to commercial product pages.
⸻
4. Trust and verification resources
This category is frequently overlooked.
Consider linking to authoritative evidence such as:
- certifications,
- standards,
- sustainability reports,
- regulatory documentation,
- product specifications,
- ingredient information,
- testing methodologies,
- corporate reports.
As AI systems increasingly synthesize claims about brands, giving machines easier access to authoritative evidence becomes strategically interesting.
⸻
5. News and current information
AI systems have to distinguish between historical and current information.
Your newsroom, press releases and corporate updates can therefore be valuable.
Just make sure the content is well maintained and properly dated.
⸻
6. Customer support
Depending on your business, this might include:
- support documentation,
- FAQs,
- returns policies,
- warranty information,
- contact information,
- troubleshooting guides.
For agentic experiences, this category may eventually become particularly important.
⸻
Should the llms.txt content be manually maintained in Sitecore?
For a small website, manually maintaining it may be completely reasonable.
For a large enterprise Sitecore implementation, I would probably avoid that approach.
Imagine maintaining 50 websites.
Or 100.
Or 500.
The moment someone changes a URL, launches a product, changes a navigation structure or retires a page, your llms.txt can become outdated.
That creates exactly the opposite of what we want.
The AI agent is now being given a curated list containing incorrect information.
For enterprise implementations, I would prefer a CMS-driven or automated model.
⸻
A better Sitecore implementation
Instead of treating llms.txt as a static file controlled by developers, create an LLM Configuration item in Sitecore.
For example:
LLM Configuration
Site Name
Site Description
Organization Links
Product Links
Expertise Links
Research Links
Trust Resources
News Links
Support Links
Optional Resources
Content editors could then manage the resources through Sitecore.
Your application generates:
/llms.txt
dynamically from those fields.
Now you have something much more scalable.
Content governance remains inside your CMS rather than inside a Git repository.
⸻
What about XM Cloud and headless Sitecore?
The implementation concept remains the same.
Sitecore’s current headless architecture supports Next.js-based applications, Headless SXA and multisite scenarios. Sitecore’s own documentation describes a multisite architecture in which hostname resolution determines which Sitecore site should be served through the application. (Sitecore Documentation)
That matters because a multisite environment should probably not have one universal llms.txt file describing every brand.
Imagine:
brand-a.com/llms.txt
and:
brand-b.com/llms.txt
Both sites happen to run through the same Sitecore/Next.js application.
The application should generate a different response based upon the resolved Sitecore site.
That gives each brand its own:
- description,
- product taxonomy,
- content resources,
- support documentation,
- market information,
- and authoritative sources.
That is much more useful than a globally shared file.
⸻
Static or dynamic?
Both approaches can work.
Static implementation
Create:
/llms.txt
as a static file deployed with your rendering application.
This is simple and low risk.
I would use this for:
- small sites,
- proof-of-concepts,
- pilots,
- relatively stable corporate websites.
Dynamic implementation
Generate llms.txt from Sitecore content.
I would favor this for:
- large enterprises,
- multisite environments,
- multilingual environments,
- websites with frequent content changes,
- organizations managing many brands.
The goal should be to make the file governable, not merely technically available.
⸻
Should llms.txt contain all of your content?
No.
This is another mistake I would avoid.
Do not turn llms.txt into a massive text dump containing every piece of content from your website.
Remember the purpose:
orientation and discovery.
Let your actual pages contain the detailed information.
Your llms.txt should tell the agent where the authoritative information lives.
⸻
And don’t create different facts for AI
This is extremely important.
You should not have:
Website says: Product lasts 8 hours.
llms.txt says: Product lasts 12 hours.
Nor should the file contain aggressive marketing claims that cannot be verified on your website.
Your AI-facing information should ultimately align with your human-facing information.
Otherwise you are introducing another source of truth into your digital ecosystem.
For an enterprise organization, that is a governance nightmare waiting to happen.
⸻
What matters more than llms.txt?
This is the part of the conversation I think gets lost.
If I had limited AEO resources, I would not start with llms.txt.
I would first ask whether AI systems can properly understand the website itself.
Google’s own current AI-search guidance reinforces this idea: traditional SEO fundamentals remain important for generative AI experiences. Google recommends ensuring crawling works, important information is available as text, internal linking is strong, page experience is good and structured data accurately reflects visible content. (Google for Developers)
My priority stack would therefore look something like this:
1. Crawlability
Can search engines and relevant AI crawlers actually access your important pages?
2. Indexability
Are canonical pages indexable?
3. Content quality
Does the website contain genuinely useful, expert-led, differentiated information?
4. Information architecture
Can machines understand the relationship between your organization, categories, products, experts and content?
5. Internal linking
Can important information actually be discovered?
6. Structured data
Are important entities and content types appropriately represented?
7. Entity consistency
Does your organization describe products, brands, people and claims consistently?
8. Freshness and provenance
Do articles contain authors, dates, expertise and appropriate sourcing?
9. External authority
Are trustworthy third-party sources discussing and validating your brand?
And then:
10. Experiment with llms.txt.
That is roughly how I would prioritize it.
⸻
What about AEO and GEO?
This is where I think the conversation gets more interesting.
AEO and GEO should not become exercises in creating hundreds of technical tricks specifically for LLMs.
Google itself now warns against many so-called generative AI optimization hacks and continues to emphasize useful, differentiated content rather than mass-producing pages or rewriting everything specifically for AI systems. (Google for Developers)
I think the better AEO question is:
How do I make my organization easier for machines to understand, verify and cite?
That involves much more than an llms.txt file.
It involves:
Entities
Who are you?
Claims
What do you say about yourself?
Evidence
Where can those claims be validated?
Expertise
Why should your organization be considered knowledgeable about the subject?
Content
Do you provide information that adds something genuinely useful?
Structure
Can machines find and understand that information?
Authority
Does the rest of the web reinforce what you say about yourself?
llms.txt potentially supports that ecosystem.
It doesn’t replace it.
⸻
Could llms.txt eventually become important?
Absolutely.
And that is one reason I would experiment with it now.
Web standards often evolve gradually.
A convention begins.
Some websites implement it.
Tools start supporting it.
Platforms experiment.
The standard either gains adoption or disappears.
The llms.txt proposal has continued evolving since its original 2024 release, and its current specification reflects broader experimentation with agent-friendly websites. (llms-txt)
We don’t yet know exactly where the convention will land.
That shouldn’t prevent experimentation.
It should simply change how much effort we invest.
⸻
So, should you add llms.txt to your Sitecore website?
Here is my recommendation.
If your website has serious SEO or content problems
No — don’t prioritize llms.txt yet.
Fix the fundamentals.
⸻
If you have a mature SEO program
Yes — experiment with it.
The cost can be relatively low.
⸻
If you manage a large Sitecore estate
Yes — but build governance around it.
Don’t manually create hundreds of disconnected text files.
Consider making llms.txt a Sitecore-managed capability.
⸻
If somebody promises a ranking boost
Be skeptical.
There is currently no credible reason to describe llms.txt as a guaranteed AEO ranking signal.
Google explicitly says it does not use the file for Search visibility. (Google for Developers)
⸻
My recommended Sitecore approach
If I were designing this for an enterprise Sitecore environment, I would implement it in phases.
Phase 1 — Pilot
Pick one website.
Create a carefully curated llms.txt.
Keep it relatively small.
Focus on authoritative resources.
⸻
Phase 2 — Validate
Test whether:
- the file returns HTTP 200,
- it is accessible without authentication,
- links resolve correctly,
- canonical URLs are used,
- no confidential information is exposed,
- URLs do not redirect unnecessarily,
- important content is not blocked,
- the output accurately reflects the website.
⸻
Phase 3 — Governance
Define who owns the file.
I would recommend joint ownership between:
SEO/AEO + Content + Digital Product
with Engineering responsible for the technical implementation.
⸻
Phase 4 — CMS integration
Create a Sitecore template allowing approved resources to be selected.
Generate the file automatically.
⸻
Phase 5 — Scale
Expand to additional brands, markets and websites.
But avoid simply cloning the same llms.txt everywhere.
Each site’s file should represent that site’s actual purpose and authoritative content.
⸻
Phase 6 — Measure
This may be the hardest part.
Do not measure success by saying:
“The file exists.”
Instead monitor:
- AI crawler activity where identifiable,
- referral traffic from AI platforms,
- brand citation visibility,
- answer-engine visibility,
- citation accuracy,
- important prompt categories,
- changes in AI responses,
- crawl behavior,
- and the quality of pages being surfaced.
Treat it as an experiment.
⸻
One final thought
I think llms.txt represents something bigger than the file itself.
For the last two decades, websites have primarily been designed for two audiences:
Humans and search engines.
We may now be entering a world with a third audience:
Agents.
An AI agent may visit your website not because a human clicked a result, but because the agent needs information to complete a task for that human.
That changes how we should think about websites.
The future may not simply be:
Human → Search Engine → Website
It may increasingly become:
Human → AI Agent → Website/Data → AI Agent → Human
That means organizations need to start thinking about whether their digital experiences are not only:
human-readable,
and
search-engine-readable,
but also increasingly:
agent-readable.
Whether llms.txt ultimately becomes the standard for solving that problem remains to be seen.
But the underlying problem is real.
So if you run a mature Sitecore website, I wouldn’t rush to install llms.txt because someone called it the latest AEO hack.
I would do something more deliberate.
Experiment with it. Govern it. Measure it.
But continue investing most heavily in the things that matter regardless of which AI platform wins:
Great content. Clear information architecture. Strong technical foundations. Structured information. Trusted claims. Demonstrable expertise.
Because ultimately, making your website easier for AI to understand begins with making your organization easier to understand.
