Hi everyone! First-time poster here, and fairly new to working with OpenAI’s APIs. I’d really appreciate any guidance or recommendations on my current project.
Current Setup
I’m working with the Assistant API 2.0 Beta in C# (using the official OpenAI C# Library) to build an AI implemenation that:
- Needs to work with specific domain knowledge (~35 000 tokens)
- Must output structured data (specifically a JSON schema array)
- Performs classification tasks (more complex than basic sentiment analysis, with multiple categories and reasoning)
- The output and input itself and the instructions of the assistant aren’t that long ((~ 125 tokens for one classification out, about the same for the input + a picture) and ~1200 tokens for the instructions)
Current Implementation
Since file search doesn’t work in combination with a structured output, I’m currently using a function call to provide the necessary domain knowledge to the model. While this works ok, I’m seeing escalating token costs as conversations and the number of threads grow. The plan is to do like a thousand classifcations a day.
The Challenge
I’m looking to optimize this setup and reduce redundancies / bring down cost in the long run. I’m considering two main approaches:
- Fine-tuning GPT-4 specifically for / with my JSON array output and domain knowledge
- Implementing a RAG (Retrieval-Augmented Generation) solution
Questions
- For my specific use case (classification with structured output), would RAG or fine-tuning be more suitable?
- Is it correct that GPT-4 fine-tuning isn’t currently supported in the Assistants API 2.0 Beta? (I’ve seen mentions that only GPT-3.5 Turbo is supported)
- How complex/time-consuming is implementing a RAG solution typically? I am worried that it is going to be really complex and time consuming.
- If fine-tuning is the better approach, how would you structure the examples to achieve both the JSON array output format and incorporate the domain knowledge?
I’m happy for any pointers, recommendations, or insights from those with more experience. Thank you so much in advance for your help and time!