Reduce in-app AI cost without reducing quality

API
1

Hi! I’ve been building with Codex for about 18 months and could use help with one issue.

I have an in-app AI with great output quality, but the API cost is too high. The main issue seems to be large tool schemas, around 67–70KB, plus repeated tool-call loops per user action.

I’ve tried smaller tool packs and prompt cache keys, but the end-to-end token cost is still high.

Has anyone solved this in a production app with tool calling? I’d love advice on reducing cost without hurting AI quality.

2

Is this a public tool? Can you limit access to paying customers? Or is this an internal company tool?

Imho only the very biggest models are reliable enough (though even 4.1 without reasoning is great for chatbots).

If you are offering a free chatbot to anyone who wants to use it I think you may need to look at semantic routing or similar to limit access to the core chatbot loop.

You can also consider some way to ration use by low value customers.

3

Divide and conquer…

I would definitely question whyyou need the huge payloads like this. Just hearing the size and problem description, I feel like the underlying process is not defined enough to make it simpler, more deterministic, and potentially on some steps, executable by cheaper models or even good old code.

Here is an example:

From what I read you use full potential model with all schemas for all tools in loop which basically drains your tokens.

But if you look closer at that process, here is what you may see:

  1. You bring some input in
  2. Based on that input something decides what tool to use
  3. Once the tool is selected something four months the tool request
  4. The request is sent for tool execution
  5. The response is collected
  6. The response is verified for conformity
  7. The sanitized response is passed further down the pipe.

So instead of sending all contacts plus all tools to highly expensive model,

  1. you extract sub process of selecting the tool by describing your tools in plain language like tool name and short description which you present with your input to a cheaper model which will make the only decision about what tool to use (this may be even fine tuned, but I don’t think the task is complicated enough for that).
  2. Once you have the tool name you take the same input and call the tool formatting model (again it can be way cheaper because you don’t need a huge model for that) that would return you the exact call to make.
  3. Value rumble tool and use code to validate the response against the schema.
  4. If the validation fails you retry (potentially switching the model to a more capable one)
  5. Once you have the clean response, you pass it further down the pipe (the step 7 from above).

Way less tokens, cheaper models, faster responses, if the quality falls on cheaper models fine-tuning will save you.

Then, you just do your math.

4

A 67–70KB tool schema is likely the main culprit here. In production, we’ve seen the biggest cost reductions come from dynamically loading only the tools relevant to the user’s current task instead of sending the entire tool catalog with every request. If a user is performing one action, there’s usually no reason for the model to see dozens of unrelated tools.

I’d also look closely at the tool-call loops. If the model is repeatedly calling tools to gather information that could be retrieved in a single step or handled through application logic, that’s where costs can spiral quickly. A lightweight routing layer that selects the right tool first, combined with aggressive caching of tool outputs, can dramatically cut token usage while keeping response quality intact.

My guess is that your issue isn’t the model quality it’s the amount of context and schema data being resent on every interaction.

5

Yeah. This also makes it easier for the model to pick the right tool.

6

This is the thing most under-defined agents miss: put as much logic as you can into the code and use LLM only where it brings the value.

7

Perhaps it’s better to prefix fixed content to avoid tool calls, thus ensuring that most content is cached.

However, I’m not sure how your agent works. if its tool calls are just fetching existing content, then this approach would work.