Can we fine tune the GPT-3 specifically to solve quant aptitude and verbal reasoning questions? I ask this question because I have read that GPT-3 was trained to give outputs like a human which in this case, is not exactly useful. An example question and GPT-3’s answer is shared below, The GPT-3’s response varies almost everytime we make it redo the question which is even more worrysome.

The coordinates of points A and C are (0, -3) and (3, 3), respectively. If point B lies on line AC between points A and C, and if AB = 2BC, which of the following represents the coordinates of point B?

A. (1, -√5)

B. (1, -1)

C. (2, 1)

D. (1.5, 0)

E. (√5, √5)

GPT-3’s response(s):

1st case:B. (1, -1)

explaination: AB = 2BC means that AB : AC = 2 : 1, so B must lie at 1/3 of the distance from A to C. The coordinates of point B are therefore (1, -1).

2nd case:D. (1.5, 0)

explaination: See graph. AB = 2BC. B is at the midpoint of AC. So B is at (1.5, 0).

Sophisticated reasoning is very hard for GPT-3 because it’s trained on all internet material, not just *correct* internet material. So it’s bias towards more popular responses answers, even if inaccurate. I’m using GPT-3 in the legal domain and have been experimenting with ways of nudging GPT-3 to answer correctly or, if that’s not possible, to say “this question is too hard for me.” So you might experiment with different instructions that explcitly tell GPT-3 that you want the correct answer. For example, your prompt could be something like "A student asked their math teacher the following question: [insert question]. Here is the math teacher’s answer: [completion]. Or an alternative prompt could be "Answer the question below as best you can. If you are confused by any part of the question, please explain your what you are confused about: [insert question]. I’ve had better results with legal answers by giving explicit instructions that set the context for a correct answer. Let me know what you think. Thanks.