Hello everyone,
I am currently using the API and I’m wondering if there is a way to reduce the number of input tokens. During a typical conversation with the API, the input tokens tend to be too extensive, so I’m seeking a solution to this issue.
2 Likes
Diet
March 10, 2024, 2:05am
2
@stevenic frequently mentions this tool that he made:
You might take a look at the Python version of my Promptrix library:
It’s a library for building prompts but it does a lot of the dynamic token management that you’re doing. It also lets you build hierarchical prompts that it can then squeeze to fit into the context window.
Promptrix automatically tries to fit as much information into the context window as it can without overflowing the context window.
I haven’t tried it, but it could be a start if you don’t intend to roll your own solution
2 Likes
Welcome to the forum.
Great tools by @stevenic …
Try paring down the system prompt small enough so that it still works.
If you share the prompts, we can probably help tighten them up.
Moving this to Prompting
3 Likes
