Your Website Is Invisible to AI Search — llms.txt Fixes That

#

People don’t Google things the way they used to. Instead of typing keywords into a search bar and scanning ten blue links, millions now ask ChatGPT, Perplexity, or Claude directly. These AI-powered search engines read your content, summarize it, and present answers — often without the user ever clicking through to your site.

That shift changes everything about SEO. Traditional optimization targets human readers and Google’s crawlers. But there’s a third audience now: large language models (LLMs) parsing your website to understand what you offer. And they’re struggling.

Why? Because most websites are messy. JavaScript-rendered navigation, pop-ups, cookie banners, ad blocks, nested divs, infinite scroll — all the things that make modern web design “work” create noise that buries your actual content. Search bots handle this well enough. LLMs don’t.

Enter llms.txt — a plain-text file that gives AI a clean, structured entry point to your website.

What Exactly Is llms.txt?

#

llms.txt is a Markdown-formatted plain-text file placed at your website’s root:

https://yourdomain.com/llms.txt

It acts as a structured summary of your site — purpose, key sections, important pages — written in a format LLMs can understand at a glance. Think of it as a README file for your website, but authored for machines instead of developers.

It was proposed in September 2024 by Jeremy Howard, co-founder of fast.ai, and documented at llmstxt.org. The concept is simple: give AI a single, reliable endpoint to understand your entire site without crawling through dozens of pages heavy with layout markup.

llms.txt vs llms-full.txt

#

There are two variants:

For most blogs and content sites, llms.txt alone is sufficient. You provide the navigation structure and let the LLM fetch specific pages when it needs full details. Reserve llms-full.txt for sites that want to be completely self-contained for offline AI processing or contexts where the model won’t follow links.

Why llms.txt Exists: The Problem It Solves

#

The web was built for two audiences:

1. Humans (rendering browsers)
2. Search crawlers (indexing for Google/Bing)

LLMs are now a third audience, and they have fundamentally different needs:

• They parse text sequentially, not visually
• They have token budgets (context windows)
• They can’t execute JavaScript
• They lose information when content is buried in layout code

A typical blog post might have:

• 200 lines of actual content
• 800 lines of HTML for header, footer, nav, sidebar, comment form

An LLM trying to extract the article sees all 1,000 lines. It wastes tokens on navigation elements, fails to identify the core message, and may misrepresent your site’s purpose entirely.

llms.txt eliminates that friction. It says: “Here’s what this site is. Here’s what matters. Here’s where to find it.” No guesswork required.

Benefits of Adding llms.txt to Your Site

#

1. Better Representation in AI-Powered Search

#

Perplexity cites sources explicitly. Other AI search tools reference the sites they pull from. If an LLM can’t confidently understand your content, it won’t cite you — and your competitors will take that visibility instead.

2. Traffic from AI Users

#

When AI search engines recommend your site, users click through. This is a growing traffic channel: Perplexity alone serves billions of queries. Sites that are legible to LLMs capture a disproportionate share of this traffic.

3. Cleaner Context for AI Tools

#

Beyond search, LLMs are used everywhere: coding assistants answering domain questions, research tools summarizing content, agents booking services. llms.txt ensures these tools get your site right.

4. Future-Proofing

#

AI agents and LLM-based browsing are accelerating. Having llms.txt now means you’re ready when this becomes standard. Most sites don’t have one yet — early adopters gain a competitive edge.

5. Zero Negative Impact

#

An llms.txt file doesn’t affect your existing SEO, Google rankings, or site performance. It’s additive. If it doesn’t help, it does no harm.

What About the Downsides?

#

Honest assessment:

• Adoption is still niche. Not all LLM tools consume llms.txt today. It’s not a guarantee — it’s preparation.
• It requires maintenance. Every few months, review and update it as your content evolves.
• No enforcement mechanism. AI tools might ignore it. But major ones increasingly don’t.
• Privacy considerations. You’re explicitly listing your site’s structure. If you have pages you don’t want AI to index, don’t include them in llms.txt.

Despite these, the risk-reward ratio is excellent. A small effort for meaningful visibility gains.

How to Create llms.txt for Your Hugo Site

#

Building one is straightforward. Here’s the full process from scratch.

Step 1: Understand the Format

#

The file follows Markdown conventions with a specific structure:

# Your Site Name

> A concise description of what this site is about.

## Section Title

- [Link Label](https://yourdomain.com/page/): One-line description

## Another Section

- [Another Article](https://yourdomain.com/another/): Brief note

Structure breakdown:

• # H1 — Site or project name
• > Blockquote — One-sentence site description (LLMs use this as context)
• ## H2 — Section headings (e.g., “Blog Posts”, “Documentation”, “About”)
• - [Markdown links] — Bullet list of key pages with one-line descriptions

Step 2: Plan Your Content

#

Don’t list every page. Curate:

• 10–30 key articles that represent your best content
• Core pages (about, contact, main services)
• Group content into logical sections

Aim for 100–500 lines total. Enough to be comprehensive, short enough to respect token budgets.

Step 3: Write the File

#

Create static/llms.txt in your Hugo project. Here’s a template:

# Zynji Blog

> A blog about web development, AI tools, and static site optimization.

## Guides

- [llms.txt AI SEO Guide](https://zynji.my.id/llms-txt-ai-seo-guide/): How to create llms.txt for better AI search visibility
- [Hugo Performance Tuning](https://zynji.my.id/hugo-performance/): Optimizing Hugo static sites for speed

## Tutorials

- [Setting Up Custom Domains](https://zynji.my.id/custom-domains/): Connect your domain to static hosting
- [Markdown Best Practices](https://zynji.my.id/markdown-guide/): Writing content that renders perfectly

## About

- [About Zynji](https://zynji.my.id/about/): What this blog covers and who writes it

Step 4: Deploy to Hugo’s Static Folder

#

Place the file in your Hugo project’s static/ directory:

your-hugo-site/
├── content/
├── layouts/
├── static/
│   └── llms.txt        ← here
├── hugo.toml
└── ...

Hugo copies everything in static/ directly to the site root during build. After deployment, verify it’s accessible:

https://yourdomain.com/llms.txt

Step 5: Advanced — Automate with Hugo Templates

#

For sites with frequent content changes, manually maintaining llms.txt becomes tedious. Hugo supports custom output formats. Add this to your hugo.toml:

[outputFormats.LLMSTxt]
mediaType = "text/plain"
baseName = "llms"
isPlainText = true

Then create layouts/index.llms.txt:

{{ .Content }}
# {{ .Site.Title }}

> {{ .Site.Params.description }}

## Latest Posts
{{ range first 10 .Site.RegularPages }}
- [{{ .Title }}]({{ .Permalink }}): {{ .Summary | truncate 100 }}
{{ end }}

## Categories
{{ range .Site.Taxonomies.categories }}
- [{{ .Page.Title }}]({{ .Page.Permalink }}): {{ .Count }} articles
{{ end }}

This auto-generates llms.txt on every build, pulling your latest content dynamically.

Step 6: Keep It Updated

#

Review your llms.txt every 1–3 months:

• Add new key articles
• Remove outdated content
• Update descriptions if page focus changes
• Verify all links still resolve

llms.txt vs robots.txt: Common Confusion

#

These are completely different files serving different purposes:

Having robots.txt does not replace llms.txt. They complement each other.

FAQ

#

Will llms.txt improve my Google search rankings?

#

No direct impact. Google’s crawlers don’t use llms.txt for ranking. Its value is in AI search engines (Perplexity, AI Overviews, etc.) and LLM-based tools. Traditional SEO and AI SEO are separate channels.

Do I need technical skills to create one?

#

No. It’s a plain text file in Markdown format. If you can write a README, you can write llms.txt. The Hugo automation is optional — manual creation works fine for most sites.

How many pages should I list?

#

Aim for 10–30 key pages. Focus on your best, most representative content. Listing every blog post dilutes the signal. Quality over quantity.

Should I create llms-full.txt too?

#

Only if you have a specific need for it. For blogs and standard content sites, llms.txt is enough. llms-full.txt is useful for documentation sites or contexts where the LLM won’t follow links back to your site.

What if I ignore llms.txt?

#

Your site still works. But as AI search grows, you’ll miss an increasingly important visibility channel. The file takes minutes to create and costs nothing.

Final Take

#

AI search isn’t theoretical anymore. Millions of queries daily go through Perplexity, ChatGPT Search, and similar tools. These systems need to understand your site — and right now, most sites give them a mess of JavaScript and layout markup to parse.

llms.txt is the simplest SEO upgrade you can make for the AI era. One file. Thirty minutes. Zero downsides.

Create yours today.

Word count: ~1,450 | Reading time: ~6 minutes