Teaching assitant to work with third party sdk

Looking for advice on how to proceed.
I want to use assistant to help my users put together scripts that use my custom sdk. Scripts generate files with a very special format. SDK methods process user data and put it in the right places within the file.
In general assistant works but definitely can be improved. What I’ve done:

  1. Fine tuned the gpt 3.5-turbo-0125 using code snippets. Used 3 prompt-answer examples per each snippet. Total number of examples came to about 600.
  2. Attach a pdf with description of every method and code snippet examples.

Problems: assistant does not give consistent answers. It is imagining methods that don’t exist and avoids using some system functions. For example I have a function to retrieve a user file but assistant tends not to use it and imagine how a file can be uploaded.
So far best results are when I add “use xxx sdk” to every example in dataset and add the suffix to user prompt: “Check attached file for available methods. Use xxx sdk”. And still results are not ideal.

1 Like