The Content-Type Contract: Serving Markdown vs HTML to AI Agents

The structure of the web was built for human consumption. For the last twenty-five years, optimizing for search engines meant ensuring that an automated crawler could extract meaning from a DOM built primarily for human eyes. We used microformats, schema.org JSON-LD, and semantic HTML tags to drop breadcrumbs for Googlebot amidst massive JavaScript bundles and deeply nested <div> soup.

In the era of Answer Engines and AI Agents, this approach is fundamentally broken.

Agents do not “see” your website. They do not care about your CSS, your interactive islands, or your highly optimized WebGL background. When an LLM crawler hits your page, it wants one thing: pure, parseable information. Feeding an LLM a megabyte of React-hydrated HTML when all it needs is the raw text is inefficient, computationally expensive, and degrades the quality of the answer it generates about your brand.

The solution is not to try and make your HTML simpler. The solution is to stop serving HTML to machines altogether.

The Content-Type Contract

HTTP was designed to solve this exact problem decades ago through Content Negotiation.

When a user agent requests a URL, it sends an Accept header detailing the MIME types it can process. Browsers typically send text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8.

When an AI crawler requests a page, we have a unique opportunity. If the crawler announces that it prefers text/markdown or text/plain, we can bypass the entire HTML rendering pipeline and serve the raw, unstyled content of the page directly.

This is the Content-Type Contract. The URL remains exactly the same. The resource is conceptually identical. But the representation changes based on who is asking.

Implementing Content Negotiation at the Edge

Let’s look at how this is implemented in practice using an Edge Worker or middleware. Instead of generating a new URL structure for your markdown files, we intercept the request at the edge:

// Example Cloudflare Worker / Middleware
export async function onRequest(context) {
  const { request } = context;
  const acceptHeader = request.headers.get('Accept') || '';

  // If the agent specifically requests markdown or plain text
  if (acceptHeader.includes('text/markdown') || acceptHeader.includes('text/plain')) {
    const url = new URL(request.url);
    
    // Rewrite the request to fetch the raw .md variant from your origin/storage
    url.pathname = `${url.pathname.replace(/\/$/, '')}.md`;
    
    const response = await fetch(url, request);
    
    // We must instruct the cache that this response varies based on the Accept header
    const modifiedResponse = new Response(response.body, response);
    modifiedResponse.headers.set('Vary', 'Accept');
    modifiedResponse.headers.set('Content-Type', 'text/markdown; charset=utf-8');
    
    // Let the crawler know the canonical location of this variant
    modifiedResponse.headers.set('Content-Location', url.pathname);
    
    return modifiedResponse;
  }

  // Otherwise, proceed to render the standard HTML page
  return context.next();
}

This architecture provides three massive benefits:

1. Zero URL fragmentation: You don’t have to maintain separate /docs and /raw-docs URLs. https://yoursite.com/about/ is the single canonical URL for everyone.
2. Infinite scalability: Generating markdown is practically free. You bypass your templating engine, your component hydration, and your layout shifts.
3. Pristine context: The LLM receives 100% signal and 0% noise. Your token count drops dramatically, ensuring the model’s context window is filled entirely with your actual content, not your class names.

The rise of llms.txt

Content negotiation solves the per-page problem. But how does an agent discover the layout of your entire site without crawling every HTML link?

Enter llms.txt.

Proposed as a standard extension of the /robots.txt philosophy, llms.txt (and its heavier sibling, llms-full.txt) acts as a machine-readable index designed specifically for Large Language Models.

Unlike an XML sitemap—which is just a list of URLs and timestamps—llms.txt provides immediate semantic context.

Here is what a well-structured llms.txt looks like:

# Web Specification Studio

> Practical documentation and standards for high-performance web engineering.

## Architecture
- [First-Party Server Tagging](/blog/first-party-server-tagging.md): Why ITP and ad-blockers make server-side tagging mandatory.
- [Structuring for AI](/blog/structuring-for-ai.md): How to prepare your DOM for Answer Engines.

## Performance
- [Edge Personalization](/blog/edge-personalization-cls.md): Stopping Cumulative Layout Shift at the edge.

Why not just use sitemap.xml?

A sitemap tells a crawler where to go. llms.txt tells a crawler what it will find when it gets there.

When an LLM is building an answer in real-time for a user (e.g., in Perplexity or ChatGPT), it does not have the time to recursively crawl your XML sitemap, download 50 pages, parse the HTML, and figure out which one contains the answer.

By fetching llms.txt first, the agent can instantly read the titles and summaries of your entire documentation tree in a single, cheap HTTP request. It can then use its limited token budget to confidently request exactly the two or three markdown variants it needs to formulate a perfect answer.

llms-full.txt: The One-Shot Context Injection

For highly specialized sites, documentation hubs, or API references, we recommend going a step further and implementing llms-full.txt.

This endpoint serves the entirety of your site’s content concatenated into a single, massive Markdown file.

While this sounds extreme, consider the use case: A developer is using an AI coding assistant (like GitHub Copilot or Cursor) and needs to write code using your proprietary framework. If they can drop the URL to your llms-full.txt into their agent, the agent instantly ingests your entire documentation suite in a single shot.

No crawling. No scraping. Immediate, perfect context.

Securing the Machine-Readable Web

When you open up pure data endpoints, you must consider the security and integrity of that data. If an AI agent hallucinates based on tampered data, the liability often falls on the data provider.

At Web Specification Studio, we enforce strict RFC 9530 integrity fields on all our Markdown responses.

When you fetch one of our .md endpoints, the response headers include:

Content-Digest: sha-256=:<base64>:
Repr-Digest:    sha-256=:<base64>:

This cryptographic signature guarantees to the consuming agent that the markdown they are processing is the exact, untampered canonical source generated by our build pipeline. It prevents Man-in-the-Middle attacks from injecting malicious prompts or SEO spam into the text before it reaches the LLM.

Conclusion

We are moving rapidly away from a web built exclusively for humans, toward a hybrid web where the primary consumer of your content is likely an AI agent working on a human’s behalf.

Clinging to HTML as your only output format is a strategic error. By embracing Content Negotiation, llms.txt, and pure Markdown endpoints, you ensure that your site is not just accessible, but highly legible to the machines that will increasingly define how your business is discovered.

Related posts

• Structuring for AI Crawlers: What Actually Works, and What llms.txt Doesn't Prove
• Data Table Serialization: Engineering HTML for LLM Ingestion