Optimizing Input Token Usage in API Conversations

[Diet]

@stevenic frequently mentions this tool that he made:

Kruel.ai V5.0 - Api companion with full understanding running 16k thanks to advanced memory system

You might take a look at the Python version of my Promptrix library: It’s a library for building prompts but it does a lot of the dynamic token management that you’re doing. It also lets you build hierarchical prompts that it can then squeeze to fit into the context window. Promptrix automatically tries to fit as much information into the context window as it can without overflowing the context window.

GitHub - Stevenic/promptrix-py: Promptrix is a prompt layout engine for Large...

GitHub - Stevenic/promptrix: Promptrix is a prompt layout engine for Large...

I haven’t tried it, but it could be a start if you don’t intend to roll your own solution