There is an example in the Evaluations guide how to generate a dataset from real traffic from completions by using the store: true parameter.
Is there a way to generate a dataset from real traffic to Assisnatnt API?
I haven’t found anything similar to the store: true parameter in the Assistant API reference. Also, I haven’t seen any API to create dataset items manually.
My use case: I am building an assistant based on the Assitant API and I want to generate a dataset from real traffic. The dataset should contain a system prompt, a history of chat messages, and attached functions for every call to Assistant API so I can use this dataset in evals and fine-tuning.
A full dump of all the API calls that Assistants make would be undesirable to OpenAI to provide. Heck, they didn’t even let you see how much it was costing when it was released.
If they managed to keep fine-tuning data locked up in some uninspectable format to train on, that could only be generated by running models that can already do what you want to do, would you even want to just accept that anyway?