GPT-3 gives bad results with Quant Aptitude and verbal reasoning questions. Can we improve it?

Prompting
1

Can we fine tune the GPT-3 specifically to solve quant aptitude and verbal reasoning questions? I ask this question because I have read that GPT-3 was trained to give outputs like a human which in this case, is not exactly useful. An example question and GPT-3’s answer is shared below, The GPT-3’s response varies almost everytime we make it redo the question which is even more worrysome.

The coordinates of points A and C are (0, -3) and (3, 3), respectively. If point B lies on line AC between points A and C, and if AB = 2BC, which of the following represents the coordinates of point B?

A. (1, -√5)
B. (1, -1)
C. (2, 1)
D. (1.5, 0)
E. (√5, √5)

GPT-3’s response(s):
1st case:B. (1, -1)
explaination: AB = 2BC means that AB : AC = 2 : 1, so B must lie at 1/3 of the distance from A to C. The coordinates of point B are therefore (1, -1).
2nd case:D. (1.5, 0)
explaination: See graph. AB = 2BC. B is at the midpoint of AC. So B is at (1.5, 0).

2

Sophisticated reasoning is very hard for GPT-3 because it’s trained on all internet material, not just correct internet material. So it’s bias towards more popular responses answers, even if inaccurate. I’m using GPT-3 in the legal domain and have been experimenting with ways of nudging GPT-3 to answer correctly or, if that’s not possible, to say “this question is too hard for me.” So you might experiment with different instructions that explcitly tell GPT-3 that you want the correct answer. For example, your prompt could be something like "A student asked their math teacher the following question: [insert question]. Here is the math teacher’s answer: [completion]. Or an alternative prompt could be "Answer the question below as best you can. If you are confused by any part of the question, please explain your what you are confused about: [insert question]. I’ve had better results with legal answers by giving explicit instructions that set the context for a correct answer. Let me know what you think. Thanks.

3

Its not perfect but you can do a lot better by just asking it to explain its reasoing.

The coordinates of points A and C are (0, -3) and (3, 3), respectively. If point B lies on line AC between points A and C, and if AB = 2BC, which of the following represents the coordinates of point B?
A. (1, -√5)
B. (1, -1)
C. (2, 1)
D. (1.5, 0)
E. (√5, √5)

Explain your reasoning step by step:

GPT-3 Output
Since point B lies on the line AC, it must satisfy the equation for a line in slope-intercept form:

y = mx + b

where m is the slope of the line and b is the y-intercept. We can plug in the coordinates of points A and C to solve for m and b:

y = mx + b

-3 = 0m + b

3 = 3m + b

Solving for m and b, we get:

m = 1

b = -3

Therefore, the equation for line AC is:

y = x - 3

Now, since AB = 2BC, we know that the length of AB must be twice the length of BC. We can calculate the length of AB and BC using the distance formula:

AB = √((x2 - x1)2 + (y2 - y1)2)

BC = √((x3 - x2)2 + (y3 - y2)2)

AB = √((32 - 02)2 + (-3 - 3)2)

AB = √(9 + 9)

AB = √18

AB = 3√2

BC = √((32 - 02)2 + (3 - 3)2)

BC = √(9 + 0)

BC = 3

Therefore, we know that the length of AB is three times the length of BC. This means that the coordinates of point B must be located one third of the way between the coordinates of points A and C:

Bx = (0 + 3)/3

Bx = 1

By = (-3 + 3)/3

By = 0

The coordinates of point B are (1, 0).

4

If you ask if it’s sure, it can correct itself too:

Are you sure about that last bit?
output
The coordinates of point B are (1, -1).