Making the v2 assistant reliable

API
1

I’ve spent a few weeks now trying to create an assistant that is able to answer questions reliably (not hallucinating, no overgeneralisation, not mixing simple things up), in a physics related space.

First thing I tried, after getting the recommendation from ChatGPT, was to add “fixes” to the prompt, for things that my agent got wrong. Yet, the more I followed that path and the more I added to the prompts, the more mistakes my assistant started to make. So, I followed my intuition again and only defined the persona and which sources it may use in the system prompt and tried to keep it minimal.

Yet, it would still happily overgeneralise every now and then. It usually takes two or tree runs (inputting the same data) and I get a wrong result. When I ask it, why it provided the wrong result, it usually states that it applied a principle that applies to a related category, but that doesn’t apply to the question it was working on.

In order to fix that I went to create functions, where the assistant persists relevant data to my backend, so it could load it when needed, and would see what type of thing it is working on, so it won’t be tempted to give answers that apply only to a different type. I am not 100% done with this, but it looks promising. Also I am trying to handle mathematical calulations as far as possible in my backend.

However now that the assistant isn’t overgeneralising, it starts to make simple mistakes, like “take the longer 4m, instead of the shorter 6m piece”. It can happen that it mixes up the same thing in a different way in the next sentence. Or it says: “take the long piece” and in the next sentence it explains why the 4m (short piece) is the right one.

I am getting the feeling that I am missing something fundamental here, but can’t figure out what it is. I had great results using ChatGPT to get advise for complex questions, so I thought I would be able to create an assistant doing the same thing. But I can’t figure out how to prevent the assistant from making mistakes all the time.

Temperature is at the lowest setting: 0.01. TopP: 1.0. I also had some success with adding an instruction to double check everything to the system prompt:

Important: Always use this 2 step process:

1. Create the answer

2. Test: Before finalizing the answer, double check whether all items that were conisdered in creating the answer are identical to the items in your mental model. Make sure no generalization has been applied. In case any discrepancy has been found, restart the process. Make sure that this test has been passed successfully before presenting any answer to the user.

However, it won’t solve such simple mistakes when it takes the short piece for the longer piece for instance. I found that it can help if I add a hidden user message to the thread, reminding it that it shouldn’t switch these things up when dealing with that part, but I can’t guard against every mistake it might make beforehand.

Has anyone run into a similar issue and found a way how to fix this?

2

The fundamental issue with your usage and your posting here: you are not talking specifically about the AI model you are employing. There will be a great deal of difference between gpt-4o-mini and gpt-4.1, for example, or gpt-4, o3, gpt-5…

(AI models themselves are not referred to as “create an assistant” so much)

On the “Assistants” API (instead of “chat completions” where you should start learning basics), you have limited model choices.

API developers that have been around since when it was deployed still have the best choice you can run on the Assistants’ endpoint when you want to pay a lot, the AI model “o1”.

image
image991×446 24.5 KB

3

Thanks for the reply. I have been mostly using 4.1. and 4o so far. I tried o1 today. I didn’t notice a particular improvement, but it has been burning my credits so fast, I was shocked - even just from testing. Is there a way I can make the assistant not simplify something it shouldn’t, not overgeneralise, not mixing simple things up along the way? I am beginning to doubt that it actually is possible to create a reliable assistant, that is dealing with a complex topic - particularly room acoustics in my case. It seems like I am constantly following this pattern:

  • assistant works great, but makes a mistake at x
  • I try to find out what lead to making this mistake and add a general guideline to the system prompt, so it won’t make this type of mistake again
  • assistant works great, but makes a mistake at y
  • I try to find out what lead to making this mistake and add a general guideline to the system prompt, so it won’t make this type of mistake again
  • Then I end up with a rather large prompt or a hidden user message at the beginning to the threat (to provide further instructions), which leads to the assistant making more mistakes because of a long and possible contradicting system prompt/instructions.
  • Then I basically just start all over again. Adding things like persisting data in the backend, so the assistant has a way to retrieve the data again when needed and has a single source of truth. But ultimately I always end up in this loop over and over.

Is it even possible these days to create a reliable assistant, that deals with a complex topic?

Any resources I should be reading/checking? I am getting the feeling I am really hitting a dead end here.

4

You have a model, and then inputs, and parameters.

You can lower top_p on non-reasoning models from the default of 1.0 to something like 0.25 to prefer the model’s best predicted certainties as output.

Then it is down to a developer message “instructions”.

You have a basic fault in your instructional style: The AI produces an output word-by-word. There is no “mental model”. In what you show, you instruct to create an answer first. The AI can’t go back and change the answer - it has no internal memory where it is editing answers on the fly or contemplating (unless a reasoning model that can produce hidden explorations). Instead you’d instruct in the style, “write out how you will solve this and outline the important facts that will need to be presented, then proceed to producing your final answer.”

You can use the “optimize” in the platform site’s playground (only offered when “responses” is selected). Put in a system message, and then have it inspected for quality and resolve the conflicting language.

The AI will still be following the task the user supplies. They have to communicate effectively within the identity and behaviors of the instructions.

5

Hi and welcome to the community!

Just a quick heads-up:
We recommend building new apps with the Responses API, since the Assistants API will be deprecated soon and sunset next year. I know this means a bit of extra work now, but it will set you up for a smoother experience going forward.

Alternatively, you can also use the Chat Completions API—it takes more effort to learn, but offers the biggest rewards in flexibility and understanding.

Hope this helps, even if it’s a little off-topic!

6

Thanks. I have found the openai cookbook, which seemed to help with creating a solid prompt. And I am leaning more towards calculating things in the background which also seems to improve reliability quite a bit - yet, it seems like it still is a long way to go.

Instructing the model to create a “mental model” seemed to increase its ability to understand the room properties from the users description.

I am migrating to responses now. I was afraid to start with that as there was a deprecation announcement, saying they would announce a deadline once feature parity was achieved. So, I didn’t want to build on something that was still lacking features - but I thought I would be way faster with development back then. Encountered a lot of issues that I wasn’t expecting. But that’s to be expected when working on something new.