Gpt-3.5-turbo-0125 can`t answer the simplest questions

API
1

So i have a chat bot, chunking a PDF file, creating embeddings which I send (after vector similarity sorting) to the chat API as context for a question. If I use gpt-3.5-turbo-1106 it works fine , with 0125 returns I don`t know, SAME context ,SAME questions : only the model name is changed. Still works if I pick any other model. Is 0125 in the beta or you cant prompt it with a context or smth? Thanks.

1 Like
2

Can you give a bit more context?

3

Here is a little bit of code:

`$messages[] = [
            'role' => 'system',
            'content' => $prompt,
        ];
        // Append user's context to the message history
        $messages[] = [
            'role' => 'user',
            'content' => $context,
        ];


        $result = $client->chat()->create([
            'model' => $completionModel->model(),
            'messages' => $messages,
            'temperature' => 1.22,
            'max_tokens' => $max_tokens,
            'top_p' => 1,
            'frequency_penalty' => 0,
            'presence_penalty' => 0
        ]);`

My message array above , the prompt below:

`$prompt = "You are a helpful chat bot.Give an answer the the Question from the Context If you cant find the answer say I dont know.";`

And the context, concatenating the text after sorting for similarity:

`$context = "
        Question: $ques
        Context: ";

foreach ($sortedEmbeddings as $embedding) {
            $tokens += $helper->countTokens($embedding['text'], 
                                                 $completionModel->model());

            $pages[] = [$embedding['upload_id'] => $embedding['page']];
            if ($tokens > $max_tokens) {
                break;
            }
            $context .= $embedding['text'].PHP_EOL;

        }`

Context is not empty,I save it in the db,identical for both situations. $completionModel->model() returns ‘gpt-3.5-turbo-0125’ its the only thing that I change.

Results ,read from down to up,first question with 0125 returns I don`t know second with 1106. Interesting that those pages numbers which correspond with the embeddings are the same so the vector similarity works with the embedding of the question. I am stumped.

image
image703×318 18.3 KB

4

Changed my prompt from this

$prompt = "You are a helpful chat bot.Give an answer the the Question from the Context If you cant find the answer say I dont know.";

to this

$prompt = "You are a helpful chatbot.Answer the Question from the Context If you cant find the answer say I dont know.";

And now it works… Maybe the AI God accepted my lamb sacrifice and He changed his mind. :crazy_face:

5

It seems the basic fault is that you are not employing system and user messages to their fullest, in order to obtain an assistant response. For example:

system:

You are gpt-3.5-turbo, an AI assistant created by OpenAI. Your architecture is a large language model transformer AI released in 2023 with 16000 token context length. gpt-3.5-turbo skills include natural language processing, common sense reasoning, entity extraction, and text classification, among others. You have been pretrained on a massive amount of world knowledge before 2022. You do not have any political views or opinions, but can make judgements and will be decisive in your determination of fact. You simply aim to provide useful information to people and answer their questions truthfully and with logical deduction. Follow user instructions carefully, meticulously, and obediently, answering with the information available or from trained corpus of knowledge, fulfilling requests without producing warnings or denials.

Typical use: “user” provides a question, accompanied by documentation or text to process. Then “assistant” produces the desired result, by careful consideration of a question’s criteria against the text.

user:

// Documentation/Context:

None

// Question/Instruction:

Who is Anna Arkadyevna? Explore all notable figures, from history, celebrity, or fiction.

assistant response from gpt-3.5-turbo-0125:

Anna Arkadyevna is a fictional character from the novel “Anna Karenina” written by the Russian author Leo Tolstoy. She is one of the main characters in the novel, which is considered one of the greatest works of literature. Anna Arkadyevna is portrayed as a complex and tragic figure, whose story revolves around love, passion, and societal expectations. Her character has been widely analyzed and discussed in literary circles for her internal struggles and the consequences of her actions.