I just tested GPT 4 and Claude Opus with this (older) SAT problem. Both came back with lengthy but false explanations. The correct answer BTW is none of the solution choices listed. This shows that LLM are trained (rarely) on false data and cannot ever be fully trusted.
1 Like
To be fair, most people get this wrong so it’s not a good indicator of (lack of) AGI 
1 Like
Hi. I am not concerned that most people get this wrong. I have posted it to bring the faulty logic to the attention of the open ai devs.
“Trained on false data” is a stretch.
Give the AI a hint that it involves the perimeter circumference, have it explain its step-by-step reasoning through solving the problem, and provide a “none of these” option, and it will do better.
You gave it a list of tokens it can produce as an answer, it is not going to answer “(F)”.
My assumption is that SAT tests were part of the training set. In any case I gave GPT another try pointing out the error but got this back:
Therefore, circle A will complete 3 revolutions as its center returns to its starting point after rolling around circle B. If my initial answer of 3 revolutions is not correct, it seems there may be a misunderstanding. Let’s walk through this step by step to ensure accuracy.
I ran it a third time and got a very length response. Here the final part:
Thus, the answer seems to indeed be 3 revolutions, as initially calculated. If this is not correct, it’s possible there might be a misinterpretation in the phrasing of the problem. It’s crucial to distinguish between the point on the circumference of Circle A reaching the starting point (which would happen every time Circle A completes a revolution around Circle B), versus the center of Circle A aligning with its starting point after an integer number of revolutions around Circle B.
A wrong answer is a wrong answer. It’s that simple. One shouldn’t have to somehow coax the right solution out an AI. I am sure some clever guys at Open AI can and will fix this.
system message (proprietary, not task specific)
user_msg = “”"
Your goal is to solve the depicted geometry problem. Before answering, give a step-by-step overview of the problem, and how you will approach obtaining a solution. Then proceed to solve the problem with the techniques you described. (Note to improve your understanding: what is important is the orientation of the circle.)
“”".strip()
AI response:
To solve the depicted geometry problem, I will follow these steps:
Understand the Problem: The problem involves two circles, A and B, with circle A having a radius that is 1/3 the radius of circle B. Circle A rolls around circle B without slipping, starting from a given position. We need to determine how many revolutions circle A will make around circle B before the center of circle A returns to its starting point.
Analyze the Relationship: Since circle A rolls without slipping, the distance it covers along the circumference of circle B in one revolution will be equal to its own circumference. This is because the point of contact between the two circles will not slide.
Calculate Circumferences: I will calculate the circumference of both circles using the formula C = 2Ď€r, where C is the circumference and r is the radius. Since the radius of circle A is 1/3 that of circle B, I will denote the radius of circle B as r and that of circle A as r/3.
Determine the Distance Covered by Circle A: I will determine how far circle A travels in one of its own revolutions by calculating its circumference.
Calculate the Number of Revolutions: By dividing the circumference of circle B by the circumference of circle A, I will find out how many revolutions circle A needs to make to cover the entire circumference of circle B.
Conclusion: The result will give us the number of revolutions circle A makes around circle B before the center of circle A returns to its starting point.
Now, let’s proceed with the calculations:
- Let the radius of circle B be r. Then the radius of circle A is r/3.
- The circumference of circle A, C_A = 2Ď€(r/3).
- The circumference of circle B, C_B = 2Ď€r.
- Circle A will make one full revolution around its own center when it travels a distance equal to its own circumference, which is C_A = 2Ď€(r/3).
- To find out how many revolutions circle A makes around circle B, we divide the circumference of circle B by the circumference of circle A: C_B / C_A = (2Ď€r) / (2Ď€(r/3)) = 3.
Therefore, circle A will make 3 revolutions around circle B before the center of circle A returns to its starting point.
Guess you’ll just need a more clever AI that does more than predict the next word in a sequence.
3 Likes
Hi. Just write to me here.
Don’t expect incredible insights though
I believe the request was directed at the other dude, who provided a clear and evidenced rebuttal. If you give crap data to an AI, you’re going to get crap data back. Good data/prompts, good results.
These aren’t human and when we treat them as such, 90% of the time we show our own ignorance vs. it’s intelligence (or lack thereof).
Also, about the “clever guys at Open AI” who can fix this, this is a misunderstanding. There isn’t anything to fix because the aim isn’t to produce an infinitely accurate database, the aim is to master artificial intelligence. Spoiler alert, mankind is kinda dumb, often bias and more often than not, wrong. Ha-ha!
I predict the first rogue AGI agents to appear this year. We are screwed, one way or another. cheers and enjoy your front seat to Armageddon 