Technical question: can publisher-declared content routes improve agent web retrieval?

ChatGPT Use cases and examples
1

Title: Can publisher-declared content routes improve agent web retrieval and citation fidelity?

I’m exploring an open proposal called index-ai and would value technical criticism from people building with web-enabled models and retrieval systems.

The problem I’m trying to isolate is simple: an agent can find the right website, but still has to infer the site’s structure, canonical content and relationships from a page built primarily for humans. That can mean processing a large rendered page, choosing among repeated or partial content, and relying on inference where the publisher could have declared intent.

index-ai proposes an optional publisher-side interface:

  • a discoverable manifest;

  • declared clean-content routes;

  • freshness and content-size signals;

  • optional typed relationships between content nodes.

It is not intended to replace crawling, browser use, Markdown conversion, llms.txt, sitemaps, Schema.org, or retrieval orchestration. The question is whether a small, declared interface could make publisher content more predictable for agents before they reconstruct it from HTML.

I’m specifically looking for objections:

  1. Is this a real missing contract, or just unnecessary duplication?

  2. Which failure modes would make publisher-declared routes unsafe or unreliable?

  3. Which existing web primitives should be reused instead of introducing another format?

  4. What is the minimum useful version an agent framework would realistically consume?

Specification: index-ai
Repository and discussions: GitHub - jordachmakaya/index-ai: An open specification for publishing structured, verifiable, retrieval-efficient content to AI agents. · GitHub

Concrete criticism is more useful than endorsement.

2

100% is unnecessary duplication.

Markdown is the de-facto language for models, and many web frameworks (like AstroJS) already offer solutions for writing content in markdown and rendering it in either HTML (Human) or raw (LLM).

Not entirely true. Most websites have concise, robot-friendly pages (sitemap for example) that provide the structure and information for crawlers to scan, extract, and organize information without dealing with the heavy payloads that a typical human-friendly HTML page does.

What is interesting is creating a discovery & communication protocol between agents. A “Google Search” that exposes functions and information. Google themselves are pursuing this with their “A2A” (Agent2Agent) protocol

3

you don’t need to duplicate a title here in the body of your Post when you already have a Topic title. :slight_smile: