Should You Create an LLMs.txt File for Your Website in 2024?

The LLMs.txt file is a new standard proposed by the tech community to allow webmasters to give specific instructions to language models (LLMs) that crawl their sites. Following the model of robots.txt which addresses search engines, this file aims to regulate the use of content by generative AI.

The current situation is particularly contradictory: on one hand, Google's official representatives publicly state that this file is useless and ignored by their systems. On the other hand, Google itself deployed an LLMs.txt file on its own domains at the beginning of December, creating a dissonance between the official discourse and the company's concrete actions.

This contradiction likely reveals an internal debate at Google about how to manage LLMs' access to web content. The technical teams seem to be acting differently from official communications, suggesting that Google's position on this subject is not yet fully stabilized.

• Google officially states that the LLMs.txt file is ignored by its systems
• Google has nevertheless deployed this same file on its own websites
• John Mueller's evasive response ("hmmn :-/") reveals discomfort on the subject
• This inconsistency suggests an unstabilized internal position

As an SEO expert, this situation perfectly illustrates the frequent gap between official communications and technical reality among major web players. When a company like Google deploys a technology on its own sites while publicly claiming it's useless, this generally reveals one of three situations: either an ongoing test, an anticipatory preparation for a future standard, or a divergence between different teams.

In this specific case, Google is likely preparing for an ecosystem where LLMs.txt files will become relevant, even if this isn't yet the case today. The company may be adopting a publicly cautious stance to avoid creating false expectations, while technically positioning itself for the future. John Mueller's embarrassed response suggests that he himself doesn't have a clear answer to give about this contradiction.

• To do now: Document the existence of the LLMs.txt file and understand its syntax to be ready to implement it quickly if necessary
• To monitor: The adoption of the LLMs.txt file by other major players (Microsoft, Meta, OpenAI) and the evolution of Google's official communications on the subject
• To avoid: Investing significant time in creating a complex LLMs.txt file as long as its usefulness is not confirmed by tangible evidence
• Recommended strategy: For sites with high content value (media, publishers, knowledge bases), creating a basic LLMs.txt file can be a reasonable precautionary measure without major cost
• Immediate alternative: Focus on already recognized directives (robots.txt, meta robots, X-Robots-Tag) that have a proven impact on crawling and indexing
• Technical monitoring: Set up an alert to follow official announcements on the recognition of LLMs.txt files by major AI players