I’m not the best dev out there but I have had a thought. Please feel free to correct me if I’m wrong. Excuse the generated text below, GPT was helpful in brainstorming this.
Proposal: Local-First Context Optimisation for Codex
I have been thinking about whether Codex could use more of the computing power already available on users’ machines to reduce cloud inference costs.
The proposal
Codex could include a local preprocessing and context-optimisation layer that performs tasks such as:
- repository indexing;
- semantic code search;
- dependency and symbol mapping;
- token counting and context budgeting;
- duplicate-context removal;
- generated-file exclusion;
- secret and credential detection;
- test-log summarisation;
- selection of only the most relevant files, functions and previous messages.
The cloud model would still handle the difficult reasoning, planning and code generation. The local machine would reduce the amount of unnecessary context entering the model’s prefill stage.
Why local tokenisation alone is not enough
Running the tokenizer locally would probably save very little by itself, because tokenisation is inexpensive compared with transformer inference.
The meaningful saving would come from using local token-aware processing to avoid sending unnecessary tokens at all.
For example:
Repository and session context: 100,000 tokens
Locally selected context: 35,000 tokens
Cloud input avoided: 65,000 tokens
The exact saving would depend on the task and repository, but large Codex sessions often contain generated files, repeated logs, duplicate code, irrelevant modules and accumulated conversation history.
A well-designed local context layer could potentially reduce cloud input processing materially on suitable workloads. A reasonable hypothesis to test might be a 10%–40% reduction in total input-token workload for large coding tasks, without materially reducing task quality.
Additional benefits
This approach could also provide:
- lower inference costs;
- faster request transmission;
- improved privacy through local secret redaction;
- fewer accidental uploads of irrelevant files;
- longer effective Codex sessions;
- higher usage limits without proportional infrastructure growth;
- better support for very large repositories.
Suggested architecture
Local machine
↓
Repository index and symbol graph
↓
Semantic retrieval
↓
Token budgeting and deduplication
↓
Secret scanning and context pruning
↓
Reduced context sent to OpenAI
↓
Frontier-model reasoning and code generation
↓
Local execution, testing and validation
A small local model could optionally assist with summarising logs, ranking files or drafting simple changes, but this would not need to be mandatory. CPU-based indexing and retrieval alone could still provide value.
Suggested experiment
OpenAI could compare the current Codex workflow with a local-first context optimiser across a sample of real repositories.
Useful measurements would include:
- original input-token count;
- optimised input-token count;
- task completion rate;
- number of follow-up prompts;
- latency;
- output quality;
- total inference cost per completed task;
- frequency of incorrectly excluded context.
The optimiser could initially operate in an observable mode, allowing users to see which files and sections were selected or excluded.
Conclusion
I do not think distributing frontier-model layers across users’ GPUs is the most practical solution. The stronger opportunity is to distribute the context-preparation workload.
Codex already executes tools, tests and file operations locally. Extending that local agent to perform intelligent, token-aware context optimisation could reduce cloud prefill demand while preserving the quality and security benefits of centralised frontier inference.
I would be interested to hear whether Codex already performs parts of this process locally and whether deeper local context optimisation is being considered.