The 5 Pillars of Technical Implementation

Lesson 3.1: Pillar 1 — Crawler Control (robots.txt for AI)

The robots.txt has been the standard way to tell web crawlers which areas they may visit for decades. What has changed: A whole new generation of AI crawlers must be explicitly addressed.

The Most Important AI Crawlers

Crawler	Operator	Function
GPTBot	OpenAI	Training crawler (no live browsing)
ChatGPT-User	OpenAI	Real-time browsing in chat sessions
OAI-SearchBot	OpenAI	Search index for ChatGPT Search — critical for citations
ClaudeBot	Anthropic	Training crawler (live access: Claude-User, Claude-SearchBot)
PerplexityBot	Perplexity	Real-time crawling for search queries
Google-Extended	Google	Not a crawler — a robots.txt token that controls whether Googlebot data is used for Gemini
Meta-ExternalAgent	Meta	Meta AI training

Best Practice — robots.txt Template

# AI Engine Crawlers — Allowed
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://example.com/sitemap.xml
# llms.txt: https://example.com/llms.txt

The 3 Most Common Mistakes

1. 404 on robots.txt: Per RFC 9309 this counts as "no restrictions" — crawling proceeds anyway. Without an explicit robots.txt you give up targeted control over AI bots. The real problem is 5xx errors: then bots crawl conservatively or not at all.
2. Global Disallow: /: Blocks everything, often a relic from development.
3. Network-level bot protection: Cloudflare/Akamai block bots before they reach the robots.txt.

Lesson 3.2: Pillar 2 — llms.txt — The AI Business Card

The /llms.txt is a proposed standard (llmstxt.org, Jeremy Howard, fast.ai). It provides language models with a high-density fact briefing — like a sitemap.xml for search engines, but for LLMs.

Structure of a Professional llms.txt

# Brand Name
> One-sentence mission with maximum semantic density.

## About Us
Short profile: founding year, location, industry, team size.

## Products / Services
- **Product A**: Fact-based description, USP, price range
- **Product B**: Technical specifications, results

## Expertise & Credentials
ISO 9001, awards, reference clients

## FAQ
- **What does [Product] cost?** From €X/month.
- **Who is it suitable for?** [Specific target audience].

## Contact
Website, email, location, management

The 4 Golden Rules

1. Fact density over marketing speak. "120 employees at 3 locations" instead of "a dynamic team".
2. Citable sentences. Every sentence must work 1:1 in an AI answer.
3. Consistency with the website. Contradictions → AI trusts no source.
4. Maximum 4,000 tokens (approx. 2-3 pages).

Lesson 3.3: Pillar 3 — Structured Data (JSON-LD Schema)

JSON-LD is a W3C standard and the preferred format for machine-readable, structured data. It provides an unambiguous "translation" of the page contents.

Analogy: HTML without JSON-LD is like a book without a table of contents. The information is in there somewhere, but you have to read everything.

Required Schemas

Schema Type	Purpose	Priority
Organization	Who is the company?	🔴 Required
WebSite	What is the website?	🔴 Required
BreadcrumbList	Page structure	🔴 Required
FAQPage	Common questions	🔴 Required
Person	Experts	🟡 High
Product / Service	Offerings	🟡 High

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Example Corp",
  "url": "https://example.com",
  "foundingDate": "2005-03-15",
  "numberOfEmployees": { "@type": "QuantitativeValue", "value": 120 },
  "address": {
    "@type": "PostalAddress",
    "addressLocality": "Düsseldorf",
    "addressCountry": "DE"
  },
  "sameAs": [
    "https://linkedin.com/company/example",
    "https://en.wikipedia.org/wiki/Example_Corp"
  ]
}

Lesson 3.4: Pillar 4 — Visual GEO (Optimizing Images for AI)

Modern AI models are multimodal: GPT-4o, Gemini and Claude can understand image content. Alt texts are therefore an active AI signal.

The Alt-Text Formula

[Subject] + [Context] + [Material/Property] + [Brand]

Alt-Text	Quality
"Serum"	❌ Useless
"Vitamin C serum bottle"	⚠️ Better
"Vitamin C Glow Serum, 30ml glass vial with pipette, Example Cosmetics"	✅ Excellent

Lesson 3.5: Pillar 5 — Answer Density

AI agents preferentially extract answers from the first 50–100 words after the H1. Placing empty marketing phrases here wastes the most valuable citable space.

Bad Example (80% of all websites)

"Welcome to Example Corp. We are a dynamic company that has been working with passion and innovation for many years..."

Good Example (Answer Density optimized)

"Example Corp is the leading DACH provider of industrial filtration technology since 2005. 120 employees at 3 locations, ISO 9001 and ISO 14001 certified. HEPA-14 filters with 99.995% separation efficiency."