Title: Can publisher-declared content routes improve agent web retrieval and citation fidelity?
I’m exploring an open proposal called index-ai and would value technical criticism from people building with web-enabled models and retrieval systems.
The problem I’m trying to isolate is simple: an agent can find the right website, but still has to infer the site’s structure, canonical content and relationships from a page built primarily for humans. That can mean processing a large rendered page, choosing among repeated or partial content, and relying on inference where the publisher could have declared intent.
index-ai proposes an optional publisher-side interface:
a discoverable manifest;
declared clean-content routes;
freshness and content-size signals;
optional typed relationships between content nodes.
It is not intended to replace crawling, browser use, Markdown conversion, llms.txt, sitemaps, Schema.org, or retrieval orchestration. The question is whether a small, declared interface could make publisher content more predictable for agents before they reconstruct it from HTML.
I’m specifically looking for objections:
Is this a real missing contract, or just unnecessary duplication?
Which failure modes would make publisher-declared routes unsafe or unreliable?
Which existing web primitives should be reused instead of introducing another format?
What is the minimum useful version an agent framework would realistically consume?
Markdown is the de-facto language for models, and many web frameworks (like AstroJS) already offer solutions for writing content in markdown and rendering it in either HTML (Human) or raw (LLM).
Not entirely true. Most websites have concise, robot-friendly pages (sitemap for example) that provide the structure and information for crawlers to scan, extract, and organize information without dealing with the heavy payloads that a typical human-friendly HTML page does.
What is interesting is creating a discovery & communication protocol between agents. A “Google Search” that exposes functions and information. Google themselves are pursuing this with their “A2A” (Agent2Agent) protocol
I run something close to this on a public archive of about 700 cited entries: llms.txt, .md/.json siblings for every page, a small read-only query API (OpenAPI plus RFC 9727 catalog) and typed cross-references. I tested it with 108 agent runs: 27 checkable questions over four routes, using LLM agents that had only a fetch tool.
Missing contract or duplication? As a set of routes, mostly duplication. Accuracy was the same on every route (107 of 108 runs correct). Only the cost differed: about 50 KB per task via JSON/API, 105 KB via Markdown, 266 KB via HTML. Every change in score came from meaning, not location. Agents read computed “related entries” as editorial links, and they read an empty list as “nothing exists”. That’s where I’d put the contract: completeness, what empty means, and where a relation comes from.
Failure modes:
Drift between representations. I once served HTML and llms.txt from builds nine hours apart, so the two showed different entry counts. Put a version stamp in every file.
Truncation that looks complete. Fetch tools cut or summarise silently. State “complete” or “part” at the top and end every file with an explicit end marker.
Guessed endpoints. Agents invent things like /search?q=, so redirect them.
Cloaking risk. A clean route that says more than the HTML can’t be verified by the agent.
Reuse instead of a new format: per page, llms.txt, sitemap lastmod, ETag, RFC 9727 with OpenAPI, and JSON-LD for relations. The only thing I had to invent was the plain-language semantics.
Minimum useful version:
one clean alternate per page, linked from the HTML head
a version stamp and a size in every file
a completeness statement and an end marker
invalid query filters return HTTP 400 instead of an empty result
Caveat: one model family, one fetch tool, and only questions with checkable answers.
