Models are totally ignoring my instructions

API
,
1

I’m beyond frustrated here.
I have a simple task to complete: load a json with a list of charts with their CODE and description. Based on the user request, the assistant must call a function, read the json loaded in the vector store and return me the CODE of the appropriate chart.

This looks like being an incredible feat, no matter how I format this damn json which now looks like this

[{
‘code’: ‘chart1’,
‘name’: ‘population growth’,
‘description’: ‘the chart analyses the population growth overtime in each country since 2019’
}]

GPT keeps inventing the code, my instructions tell many times not to invent anything and use codes provided in the documentation.
Sometimes it uses correct codes, but I can’t keep trying forever for an existing code!

What am I doing wrong here? 4o mini and 4o both can’t be consistent, I can’t propose this s*** to my clients.
There must be something I’m doing wrong, how would you accomplish such task?

2

Have you tried setting the top_p and/or temperature parameters to a lower value?
Maybe this can help you, as I have seen some examples for tasks like this.

3

  1. Vector store to retrieve JSON? Who’s vector store? Embeddings is for language; you will have an overwhelming commonality of “looks like some computer code” within the embeddings with JSON being passed to an embeddings model.

  2. Is there quality semantic information, splitting of chunks by only semantically relevant kernels of knowledge, and you can inspect this return to see if high quality information is being offered to the model?

  3. Can gpt-4o-2024-05-13 or gpt-4-turbo do the task much better? You can chalk it up to a reduced-size multimodal model trained to speak in post-trained emotive voices, use ChatGPT’s tools, and chat like Marv the sarcastic AI instead of performing a developer’s tasks, and ignoring instructions because you aren’t “OpenAI” in the new model hierarchy scheme this model uses.

4

hey…this…is kinda giving some results! I’ll further test. In the meantime I converted the JSON to a text, maybe simpler for gpt…

1 Like
5

I can confirm I can consistently get results now. They are not always 100% correct but it’s incredibly much better than before. Temperature set to ZERO, top p to ZERO as well. It’s not inventing anything now.

2 Likes
6

Yes, it keep happening.

Please read more on “Prompt Engineering” !

All LLM’s are like my trainees, direct them at precise task and ask them to do it! Else, it’s a Mess!