Why Deprecating the Legacy GPT-3.5 is a bad idea | The default model can't break down problems

Prompting
1

In the DeepLearning.AI course on Prompt Engineering, it gives an example of how GPT can solve a problem and think through the answer by breaking down the problem step-by-step. The Default GPT-3.5 model is not capable of doing this, however, GPT-4 and the Legacy GPT-3.5 can solve the problem by breaking down the problem into links in a chain of steps. Note that this was originally a situation that was presented by Isa Fulford using the OpenAI API and the model was “3.5 turbo” which there doesn’t seem to be an equivalent in the web app version.

Here are the results form my own test. Please test to confirm. The following text is in Markdown format.

Problem Solving

Note that in the GPT-3.5 model the instructions seemed too complicated and it could not break down the instructions into smaller links in a chain, but the GPT-4 model was able to do this.

USER

Your task is to determine if the student’s solution is correct or not.
To solve the problem do the following:

  • First, work out your own solution to the problem.
  • Then compare your solution to the student’s solution and evaluate if the student’s solution is correct or not.
    Don’t decide if the student’s solution is correct until you have done the problem yourself.

Use the following format:
Question:

Student’s solution: <student’s solution here inside a Markdown codeblock>

Actual solution:

Is the student’s solution the same as actual solution just calculated:

Student grade:

Question:
I’m building a solar power installation and I need help working out the financials.

  • Land costs $100 per square foot
  • I can buy solar panels for $250 per square foot
  • I negotiated a contract for maintenance that will cost me a flat $100k per year, and an additional $10 per square foot
    What is the total cost for the first year of operations as a function of the number of square feet.

Student’s Solution:
Let x be the size of the installation in square feet.
Costs:

  1. Land cost: 100x
  2. Solar panel cost: 250x
  3. Maintenance cost: 100,000 + 100x
    Total cost: 100x + 250x + 100,000 + 100x = 450x + 100,000

ASSISTANT using the GPT-4 model and Legacy 3.5 model

Question: I’m building a solar power installation and I need help working out the financials.

  • Land costs $100 per square foot
  • I can buy solar panels for $250 per square foot
  • I negotiated a contract for maintenance that will cost me a flat $100k per year, and an additional $10 per square foot
    What is the total cost for the first year of operations as a function of the number of square feet.

Student’s Solution:

Let x be the size of the installation in square feet.
Costs:
1. Land cost: 100x
2. Solar panel cost: 250x
3. Maintenance cost: 100,000 + 100x
Total cost: 100x + 250x + 100,000 + 100x = 450x + 100,000

Actual Solution:

Let x be the size of the installation in square feet.
Costs:
1. Land cost: 100x
2. Solar panel cost: 250x
3. Maintenance cost: 100,000 + 10x
Total cost: 100x + 250x + 100,000 + 10x = 360x + 100,000

Is the student’s solution the same as the actual solution just calculated: No

Student grade: Incorrect

ASSISTANT using the GPT-3.5 model

Actual Solution:
Let’s assume that the size of the installation is x square feet.

  • Land cost: 100x dollars
  • Solar panel cost: 250x dollars
  • Maintenance cost: 100,000 + 10x dollars
    Therefore, the total cost for the first year of operations would be the sum of these costs, which is:
    Total cost = Land cost + Solar panel cost + Maintenance cost
    Total cost = 100x + 250x + (100,000 + 10x)
    Total cost = 350x + 100,000 + 10x
    Total cost = 360x + 100,000 dollars

Is the student’s solution the same as actual solution just calculated:
Yes

Student grade:
Correct