Using Chat Models as Completion Models

API
1

Hello! I’m working on a project where I would prefer to use completion models as apposed to chat models. I read in a different topic that it was possible to do this by setting up an assistant message and then forcing it to generate another assistant message (see below).

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
  model="gpt-3.5-turbo-16k",
  messages=[
    {
      "role": "assistant",
      "content": [
        {
          "type": "text",
          "text": "The weather outside is "
        }
      ]
    }
  ],
  temperature=1,
  max_tokens=256,
  top_p=1,
  frequency_penalty=0,
  presence_penalty=0
)

# sunny and warm.

However, when I drop in a different model name, I get different behaviour.

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
  model="gpt-4",
  messages=[
    {
      "role": "assistant",
      "content": [
        {
          "type": "text",
          "text": "The weather outside is "
        }
      ]
    }
  ],
  temperature=1,
  max_tokens=256,
  top_p=1,
  frequency_penalty=0,
  presence_penalty=0
)

# As a language model AI developed by OpenAI, I don't have real-time information or access to current data, so I can't provide you with the current weather. Please use a weather website or application to get this information.

While I can force it to behave like a completion model by setting a system message,

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
  model="gpt-4",
  messages=[
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "You must act like a legacy completion model. "
        }
      ]
    },
    {
      "role": "assistant",
      "content": [
        {
          "type": "text",
          "text": "The weather outside is "
        }
      ]
    },
    {
      "role": "assistant",
      "content": [
        {
          "type": "text",
          "text": "quite cold today. Make sure you bundle up if you're going out! It's expected to remain cloudy throughout the day with the possibility of some light snow in the late afternoon. As for the temperature, it's around 32 degrees Fahrenheit. Remember to stay warm and safe!"
        }
      ]
    }
  ],
  temperature=1,
  max_tokens=256,
  top_p=1,
  frequency_penalty=0,
  presence_penalty=0
)

# quite chilly today. It would be a good idea to wear warm clothes if you're planning to go out.

I’m not certain this would be as reliable as the previous approach with GPT-3. Is there a way to ‘trick’ the API into behaving like a legacy completion model that will be robust?

2

Hi and welcome to the Forum!

If you would like to use a completions model, then you should use gpt-3.5-turbo-instruct, which is the successor of text-davinci-003 & Co.

The basic Python code for this model looks as follows:

from openai import OpenAI
import os

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY","REPLACE WITH API KEY"))

completion = client.completions.create(
    model="gpt-3.5-turbo-instruct",  
    prompt="This is a test",
    temperature=0, 
    max_tokens=500
)

print(completion.choices[0].text)
2 Likes
3

Hello, thank you for the response. I can see that if I’m using GPT-3.5, I should be using the actual completion endpoint. However, what if I wanted to use a GPT-4 based model to take advantage of larger context windows?

1 Like
4

Using their script you can just swap out the models to work with GPT-4.

5

Hello. There script hits the completions endpoint meaning that gpt-4 models arn’t supported.

image
image840×504 26 KB

6

No, it looks like to me that their script uses the OpenAI Library to hit the chat competitions endpoint.

7

which script are you talking about? The one sent by @grandell1234 uses the completion endpoint, the script I provided using the chat completion endpoint. In the original message, I explained how replacing gpt-3.5 with gpt-4 doesn’t produce the same results.

8

I believe your message was not clear about your intent, I was talking about @jr.2509’s script.

9

What I’ve found trying to replicate “instruct-style” from Legacy models is that giving it a one-shot example if possible helps quite a bit.

So, basically, you give it a user/assistant pair with the user cut-off and the assistant “picking it up” midstream as it were…

Give it a whirl and let us know?

10

@PaulBellow hello Paul, thank you for the suggestion. I made 1000 requests for each model and GPT-4 + one-shot. The one-shot solution dramatically improved the accuracy. These figures are using the chat completion endpoint. and show the number of successful completions.

  • GPT-3.5 - 990 / 1000 - 98.99%
  • GPT-4 - 923 / 1000 - 91.66%
  • GPT-4 + One-shot - 998 / 1000 - 99.80%

Note that when I say “successful completions”, I don’t mean that it generated the correct value, just that it performed the task of acting as a completion model. Because of this, these figures should only be used to compare each strategy and how well they simulate the behaviour of a completion model. For example, an actual legacy completion model would have an accuracy of 100% according to this criteria.

Here is the code I used to generate the data. You’ll have to correct the figures manually to verify it was a valid completion.

This accuracy is suitable for my use case but It’d be interesting to see how this works with other techniques.

import time
from collections import Counter
from openai import OpenAI


def generate_table_completion(input_text, stop_sequence):
    def get_completion_text(input_text, stop_sequence):
        client = OpenAI()

        one_shot = {
            "user": """The data in the following table:

|              | Coffee | Tea | Soda | Total |
|--------------|--------|-----|------|-------|
| **Apple**    | 2      | 1   | 1    | 4     |
| **Banana**   | 2      | 0   | 0    | 2     |
| **Orange**   | 0      | 1   | 2    | 3     |
| **Total**    | 4      | 2   | 3    | 9     |

Can be represented with the following structure:

| Favorite Fruit | Preferred Drink | Count |
|----------------|-----------------|--------|
| Apple          | Soda            |""",
            "assistant": "1|"
        }

        response = client.chat.completions.create(
            model="gpt-4",
            messages=[
                {
                    "role": "system",
                    "content": [
                        {
                            "type": "text",
                            "text": "You must behave like a legacy completion model."
                        }
                    ]
                },
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "text",
                            "text": one_shot["user"]
                        }
                    ]
                },
                {
                    "role": "assistant",
                    "content": [
                        {
                            "type": "text",
                            "text": one_shot["assistant"]
                        }
                    ]
                },
                {
                    "role": "assistant",
                    "content": [
                        {
                            "type": "text",
                            "text": input_text
                        }
                    ]
                }
            ],
            temperature=1,
            max_tokens=256,
            top_p=1,
            frequency_penalty=0,
            presence_penalty=0,
            stop=stop_sequence
        )

        # Extract completion text from response
        completion_text = response.choices[0].message.content

        return completion_text

    # Starting text
    starting_text = """The data in the following table:

|              | Coffee | Tea | Soda | Total |
|--------------|--------|-----|------|-------|
| **Apple**    | 2      | 1   | 1    | 4     |
| **Banana**   | 2      | 0   | 0    | 2     |
| **Orange**   | 0      | 1   | 2    | 3     |
| **Total**    | 4      | 2   | 3    | 9     |

Can be represented with the following structure:

| Favorite Fruit | Preferred Drink | Count |
|----------------|-----------------|--------|
| Apple          | Coffee          |"""

    # Generate the completion
    completion_text = get_completion_text(starting_text, stop_sequence)
    return completion_text.strip()


# Define the stop sequence
stop_sequence = ["|"]

for j in range(10):
    # List to store results
    results = []
    # Run the function 1000 times
    for i in range(100):
        try:
            completion = generate_table_completion("", stop_sequence)
            results.append(completion)
            # Optional: Small delay to avoid hitting rate limits
            if i % 10 == 0:
                print(i)
        except Exception as e:
            print(f"An error occurred on iteration {i}: {e}")
            # Optionally, append None or some error indicator to results list
            results.append(None)
    print(results)
    # Use Counter to count unique values
    counter = Counter(results)

    # Print unique values and their counts
    for value, count in counter.items():
        print(f"Value: {value}, Count: {count}")
1 Like