Hi,
I am using chatgpt API for classification. I have to classify 2000 docs. Right now I am batching 10 docs per prompt and I am instructing the model to return a list of labels.
The issue I am facing is the model returns only 9 samples only in edge cases like in 2% of the cases leading to structural issues in data loading.
Can someone please let me know is there a better way to classify? I was thinking sending only 1 doc per request but that would mean a lot of latency and might hit ratelimits leading to more latency.