Feature Request: Managed Hot Context / Agent Memory Store
I would like an OpenAI-managed data layer designed specifically for low-latency agent memory and context retrieval.
When integrating AI into business applications, one thing that is coming up again and again is context management and speed to response. Every time I’m working on anything that handles context it feels like I am reinventing the wheel and often I need to send data over the wire that I know in advance will be needed by the LLMs anyway. It would be much faster if it was already there and optimized for AI ingestion. Seems like a great managed service offering that could be an excellent revenue source for OpenAI.
The basic idea would be to let an API customer purchase a bounded amount of persistent context storage — for example 20 GB — and continuously stream application context into it.
OpenAI would manage:
-
semantic/vector indexing
-
keyword and metadata indexing
-
structured/entity indexing where appropriate
-
recency and relevance ranking
-
deduplication
-
automatic expiration or FIFO cleanup when the storage limit is reached
-
retrieval optimized specifically for OpenAI models
The important distinction is that the entire 20 GB would not be inserted into the model context window. Instead, the store would act as an AI-native memory layer from which OpenAI automatically retrieves the small amount of information relevant to each inference.
Conceptually:
Application → continuously synced OpenAI Context Store → automatic retrieval → GPT inference
This would be especially valuable if the storage/index infrastructure could be colocated or otherwise highly optimized with OpenAI inference infrastructure, reducing the latency associated with external RAG systems and tool calls.
Example use case
We are building an automotive conversational agent.
It needs access to:
-
recent vehicle inventory
-
vehicle attributes and pricing
-
previous search results
-
customer conversation history
-
entities previously discussed
-
market observations
-
recent application events
Some queries should still reach our authoritative database, particularly analytical or transactional queries. But a very large percentage of conversational questions could be answered from a continuously updated “warm” context store.
For example:
“What was that red F-150 we looked at last week?”
or
“You mentioned a Tahoe earlier that was priced unusually well. Is it still available?”
Those questions should ideally not require us to build and operate our own:
OpenAI + vector DB + Redis + conversation DB + embedding pipeline + retrieval service + summarization pipeline + context-ranking layer.
Suggested abstraction
Something along the lines of:
OpenAI Context Store
-
Capacity: 1 GB / 20 GB / 100 GB / etc.
-
Streaming ingestion API
-
Tenant isolation
-
Configurable retention
-
Metadata and structured fields
-
Automatic semantic indexing
-
Automatic retrieval during Responses API calls
-
Explicit search API when application control is needed
-
User/entity/conversation namespaces
-
Regional/data-residency options
-
Usage and retrieval observability
Applications could attach one or more context stores to an agent/model request.
The developer should still be able to explicitly control retrieval when necessary, but the default could be OpenAI automatically determining which information is useful for the current request.
This feels like a natural evolution of persistent Conversations, File Search/vector stores, prompt caching, and agent tooling: rather than developers repeatedly rebuilding memory infrastructure around the model, OpenAI could provide an AI-native “hot memory plane” optimized for inference.
For us, this would materially simplify the architecture of production agents while potentially improving both latency and response quality.