I understand that GPT models have a context size. I am using a prompt that grows in size. I would like GPT to look at only the last/bottom part of the prompt when it gets larger than the context. I need a sort of window. How do I scroll the context size to my prompt?
There are multiple ways to do so.
-
you can ask the model to give you 4 chunks of your text - then let it summarize the first quater - which will lead to slowly forgetting what was said before
-
you can just hold the prompts and the responses in an array and do something like this (which would be more radical):
conversation = [
{"role": "user", "content": "First prompt"},
{"role": "assistant", "content": "First response"},
{"role": "user", "content": "Second prompt"},
{"role": "assistant", "content": "Second response"}
]
# Remove the first prompt and response
conversation = conversation[2:]
- Or you could save the whole conversation in an information graph and add agents to retrieve older information if needed
I’m leaning twords #2. What is the best way to measure the size of my prompt in tokens? If I’m going to cut off part of it I should have some rule for estimating its size. Also, where do I find the size of a given model’s input?
I think you are looking for tiktoken.