I use GPT4 API to return a json file with a list of n charts that GPT considers particularly insightful to analyse a given dataset.
I would like to give the user the option to resubmit the prompt to ask for a new Json with 10 or so new charts, that have to be different from those that the API has already returned.
In order to avoid repetition I add the first JSON to the second prompt, but the total token count goes up to 8500 and I get an error.
I tried to simply add a “only propose charts you have not already proposed” to the prompt, but works erratically.
I assume you have a chatbot that continues to add prior messages back as conversation history. Without management, the prior conversation in the session would continue to grow until the input, plus the max_tokens space that is reserved for an answer, is greater than the context length of the AI model.
First, ensure you are using a reasonable max_tokens value, such as 2000. This is the amount of the model’s context length that is used for forming a response. It subtracts from the amount of input you can send. Proper use of this parameter is a common misunderstanding.
Then, you will certainly need some form of management, to only pass the most recent conversation that is relevant. A reasonable approach is to measure the size of the current user question plus system prompt, add in the size of max_tokens specified, to get the remaining token space that can be used for old conversation between the user and the AI.
A library such as tiktoken can provide accurate token counting to maximize conversation history if you don’t want to simply limit conversation to the last five turns, or other simple techniques.
Unfortunately my app (mparanza.com) does not work with a user chat as you describe.
The user uploads a sales dataset. The app sends to Gpt4 the metadata of the dataset plus a set of instructions (structure of the expected Json file, number of expected charts…). There is no 'conversation in the sense that you describe.
Gpt returns the Jason file that describes the required charts (my app then plots them and renders a sales présentation.) As mentioned the prompt is about 4000 tokens. This prompt is generated by my app. Limited input by user (only the dataset)
If this first presentation is not convincing enough the user requests more charts. I need to send back the whole prompt but also the list of the charts that have already been “proposed” because I only want new ones… And I crash the 8000 token limit…
You have a complex application that is hard to deduce from the site’s documentation.
4000 tokens of just instruction?
Perhaps this is a case for not solely relying on GPT-4 to do all the thinking in one go. One concept would be dynamic prompt instructions, constructed piecemeal by the software, if you have individual instructions for each type of chart, for example. You could then assemble the partial chart prompt, listing all to be tried in the first run, by a first initial sampling of part of the supplied user data to come up with appropriate classes of charts. Then follow-ups could likewise try another set of prompt instructions, eliminating those stored in metadata. Alternately, you could just have “basic charts” and “esoteric charts” as different prompts for the user to try or stages of processing.
You are probably pushing the AI’s ability to do math to the limits if you aren’t using function calling and python sandbox to do the data processing and merging of categories, but you can look at where gpt-3.5-turbo-16k can still follow some of the steps. Calling it ten times to get a job done is the same cost as one GPT-4 call.
I only use Gpt4 to “choose” which charts I need to plot given the dataset. No calculations.
If you will it gives me the “blank charts”. Then I plot them with my app using python (40k rows of code).
Btw I have no idea of why Gpt4 is capable of choosing the right charts and thus building the structure of a sales présentation, but in practice it works similarly (more or less as well) to what a human would do. Once I have plotted the chats my app I also use GPT4 to comment them
My problem (sorry again for the confusion) is the following.
Example:
First run Gpt4 suggests these 4 charts: A, B, C, D. Maybe A is line chart by region and sales and B is a column chart by category and units sold…
I (user) plot my four charts. After examining them, I see I would need 4 charts more to understand my sales, because I want to have more visibility on price by country, for example. So I resubmit the prompt (adding the suggestion of looking into price by country) but I do not want any of the charts of the first run A B C D to be in the second response.
What I did before was to include the list of the first four charts in the second prompt, but this brought my prompt size to 8500 tokens (I ask for a Jason of 55 plots at the beginning).
So now I simply ask GPT to suggest new charts (but this does not work because I believe it has no session state memory) and delete the charts I have already seen in python.
