I tried tweaking the temperature and max_tokens or omitting them, I also tested each models but still ChatGPT gives more comprehensive answer. I’m not expecting exact similar results, I just want to get answer in API as detailed as ChatGPT. Does anyone has an explanation or solution to this? Thanks.
I think the trick with the API vs. ChatGPT, is you have to bring in context yourself with the API (history, or other related information via RAG). Whereas the history is built-in using ChatGPT.
What are the specifics of your case?
Text Analysis such as summarization, themes, sentiments, etc. I used the same prompt. My prompt is something like this:
What are the themes in this text:
document_chunk.
I know chatgpt is well formatted, i can do that on my code. I just wanted a comprehensive answer as chatgpt gave. I also tried to fine tune and bring the results from ChatGPT to training data but still doesn’t work.
What models are you using in ChatGPT? Same question for the API.
Also what are your API settings? Temp?
What is your System message in the API? (Assuming the chat endpoints here)
I am always using 3.5-turbo-1106 because the chunks are very long but I tried 3.5-turbo-16k too but still getting a short answer. I always set max_tokens to the maximum. Temperature is 1 but I tweaking it from 0.5 to 1 from time to time to get a optimum result.
However, the training loss of the fine-tuned model is 0.4.
Actually there is an instruction in the system message to format the answer. Here is the detailed prompt I am using currently using:
"What are the themes covered in this text, and the sub-points presented for or against each theme?
If the text is persuasive, an opinion or a review text, list all the major claims discussed in the text as well as the arguments, evidence, and appeals to emotion and reputation for and against each claim.
If the text is a factual, explanatory, or research article, list all the major statements discussed in the text as well as the arguments, evidence, and sometimes appeals to emotion and reputation for and against each claim. Put the answer in this format: \nTitle: Title of the text\nSummary: a short summary of the given text and only summarise the text without any bias, excluding any self-interpretation of the text.\nTheme 1: Name of the theme.\nSummary: One-sentence summary or definition of Theme 1 as stated in the text.\nArguments for:\n— x\n— y\nArguments against:\n— x\n— y\nand so on for all the themes and sub-points in the text.
I put the instruction in the system message because I am always getting successful answers than putting it along the user message.
.
Where do you find out what RAG information and history it pulls in chatgpt? I would love to not have to test it out as much as possible.