GPT-4 - The programming and math genius

Prompting
1

GTP-4 The math and basic programming genius

I am paying for GPT-4 and asked it to create me a regular expression that matches exactly 4 digits in a text followed by a hyphen with the possibility of there being spaces around the hyphen.

Yes, you guessed it - I only needed a simple regex to match year ranged like 2021-2023.

GPT-4 just not only couldn’t find a working solution, it also provided me with over 30 incorrect regular expression “solutions” (I am not exaggerated here) by only try its solution when I told it that it didn’t work.

First it matched numbers that were longer than 4 digits, a clear logical error on its part. It never checked the solution it provided if it was actually working on a sample text I also provided. I always had to manually tell it that it’s solution doesn’t work.

After about 10 incorrect solutions provided (“Oh I see that the regular expression doesn’t conform to the requirements. Let me try again” - doh!), it was finally able to come up with a regex that matched exactly 4 digits, no more and no less which is actually ridiculously easy to do but GPT-4 was bound on using word boundaries which of course won’t work and never consider any of the many other ways to accomplish this.

After it finally seemed to come up with a working solution, already knowning its limited grasp on a whole problem and instead focusing on a solution for a detail, I asked it “What if the text starts with a 4 digit number?” which I of course immediately saw that the regex was not checking for.

Same thing happened again and after providing about 10 incorrect solutions which I had to point out individually that they all didn’t work, it finally was able to come up with a working regex matching a number with exactly 4 digits.

Wow! Bravo and well done I guess. It took about 45 minutes for this. My 14 year old nephew could have solved the problem in a few minutes.

Without burdening you with the rest of the incorrect solutions it came up with to “solve” this most basic regex “problem” and 15 minutes later with no correct solution in sight, I just told it to stop since it was utterly useless to proceed.

The same thing happened with a simple mathematical equation involving 3 numbers and 1 unknown x. You would think GPT-4 would be able to provide an answer in a snap. 30 minutes of being given incorrect values for x, I finally told it to stop this embarrassment and gave it the correct value of x. After which it calculated the equation out and confirmed that my result was correct while also apologizing that it should have been able to solve this simple equation for 1st graders, yet insisting that it was still capable of solving mathematical equations. What a joke!

I have plenty of more examples of GPT-4 ridiculously low level of being capable of solving the simplest of problems.

2

Can you share the prompts you used and the results obtained? If you have a chat link that would great, thanks.

3

My 1st prompt
Create regex expression for 4 digits (a year) followed by a dash and then another 4 digits (a year).

Returned:
\d{4}-\d{4}

With an explanation of each part and examples it would match with.

I realised I missed the spaces you wanted.

My 2nd prompt:
Update the expression to allow for optional spaces on either side of the dash

Returned:
\d{4}\s*-\s*\d{4}

With more explanations and examples.

4

"create me a regular expression that matches exactly 4 digits in a text followed by a hyphen with the possibility of there being spaces around the hyphen

Yes, you guessed it - I only needed a simple regex to match year ranged like 2021-2023."

NO -I did not guess that you needed o regex for a year RANGE, because that is not what you requested! What you requested is 4 digits followed by a hyphen with possible spaces around the hyphen. That certainly is not a year range.

You get the right answer by asking the right question. And if you want a year range, you can ask tor that directly instead of trying to give (wrong) rules for it. In the OpenAI playground I simply asked:

System: You are an expert in Regex.

User: Create a regular expression that matches a year range inside any other text

Assistant: To create a regular expression that matches a year range inside any other text, you can use the following pattern:

\b(\d{4})\s*[-–]\s*(\d{4})\b

That was in first try and using the cheaper GPT-3.5.

GPT-4 gave a more complex, but also working answer also checking for first digits being 1 or 2:

(?<!\d)(\b[12]\d{3})\s*-\s*([12]\d{3}\b)(?!\d)

(with detailed explanations that are not included here)

Finally, I might add that you are sailing dangerous waters if you are relying solely on GPT to build your regexes without enough of your own understanding either on how to ask GPT the correct way or how to build even simple regexes (like that for year range) yourself. Both GPT and Regex are very powerful tools, but you really need to study both to get what you want!

5

Yes, of course I know that I can’t rely on GPT-4 answers. It doesn’t even check the solutions against the sample text I provided. This was the reason why I constantly had to say that the solution provided doesn’t work and that’s what took 45 minutes.

I was only testing GPT-4 objectively. Of course I knew the proper regex beforehand.

The problem was that GPT-4 was so sure about its first solution (using word boundaries) that it never attempted a different approach like a simple [^\d]. It was steadfast in trying to find a solution using word boundaries.

Whether or not an example of 2022-2023 is a year range is completely irrelevant. It has zero impact on the solution. I merely mentioned it to highlight how easy the problem was to solve.

But once GPT-4 has decided a way to find a solution (word boundaries in this case), it will only try to find a solution this way. It is incapable of taking a step back and reassess the problem and looking for different methods of solving the problem.

Don’t get me wrong - I have nothing against GPT-4 and it is quite amazing in other fields. By testing it’s capability in specific yet easy fields, my goal was merely to assert weather it could solve a specific problem or not. With the hope that it not, these issues would be improved in further upgrades.

6

Did you try using numbers with more than 4 digits?

GPT-4 was struggling with this, matching for example 5 digit numbers. Yes, a 5 digit number matches a 4 digit number (it simply ignores the first digit) but it doesn’t take into account that any numbers that don’t have exactly 4 digits should not be matched.

This was an ongoing problem with the solutions GPT-4 provided, namely that it tried to solve the problem with word boundaries and never evaluated after about 15 fails to fall back and attempt a different approach.

I am only pointing this out so that this issue can be addressed in future versions. GPT-4 leans heavily on the first “solution” it deems to be correct and it takes a long time for it to realize that this “first solution” just isn’t working and that it would be so much better to abandone it and reassess the problem and trying a different approach.

I’m just giving feedback here which if implemented, would make GPT-4 do much more powerful.

7

\s means any whitespace character which is horribly incorrect. I distinctly specified that it should not matter whether or not the dash is sandwiched between SPACES. Which can be accomplished as simple as this:

?-?

Trying to match any and all whitespace character is another fundamental, logical misunderstanding that should not have happened. After I pointed this out GPT-4 apologized as usual for the “oversight” of matching a whole class of characters instead of a specific, unique character - namely a simple space " ".

8

The answer you received:

(?<!\d)(\b[12]\d{3})\s*-\s*([12]\d{3}\b)(?!\d)

is incredibly bloated and unnecessarily complicated, serving no purpose whatsoever.

Just by quickly analyzing the regex provided, I can see a bunch of fundamental logical errors like matching the entire white space character class with \s instead of only matching a simple space that may or may not be there.

I can see word boundaries in this “solution” which will not work.

I see GPT-4 trying to match exactly 3 digits which makes no sense.

All in all it’s horribly bloated and without trying it I highly doubt that it would work.

Remember that \d{3} will match any 3 digit number like:

123456789

$1 = 123
$2 =456
$3 = 789

9

I see what you are trying to do in your revised solution.

Although fundamentally incorrect by matching any white space character instead of a simple space, matching zero or more characters would be yet another viable approach which GPT-4 never even considered, namely for example:

()*

10

You probably can’t see it but there is a simple space between the brackets.

11

I did specifically request only 4 digit numbers, no more or less. Only numbers with exactly 4 digits should be matched.

I can’t put it clearer than that. If GPT-4 thinks a 5 digit number qualifies as such and then only matches the trailing 4 digits, omitting the first digit of the number in the match, then there is something seriously wrong with its logical understanding.

12

I guess the quality of the result is proportionate to the quality of the prompt.

As @pasi.karhu mentioned above, you’re expecting responses to questions that haven’t been asked yet. If I add more requirements to my prompt, it satisfies them. You haven’t listed your full requirements or your prompts, so even as a human I can’t meet your expectations.

As AI improves, we’ll all have to explain less, but the current abilities were not something I expected to see in my lifetime and are improving rapidly.

13

Do you want to see the simple equation I gave GPT-4 to solve which had only three 1-digit numbers and an unknown number of x. Asking it to calculate the value of x.

Not very difficult you would think? After 30 minutes of receiving incorrect answers (of course having to point out each incorrect answer individually), after me asking, Chat-GPT finally admitted that being able to solve this simple equation was highly unlikely, me giving it the correct value of x, it confirming that it is correct and me telling it to stop any further attempts as to not waste any more computing power.

Chat-GPT-4 on follow up still insisting it was fully capable of solving such simple equations (this is what is expected of 1st graders). Fremdscham.

14

Use gpt4-v and see what it does. Take a pic of your handwritten prompt, keep it simple, and see how it works out!

15

here is example a prompt for regex

:dizzy: Topic: Regex Generator Email Address

:robot: :gear: Message From ChatGPT (System):

You are an assistant that Regex Generator Email Address

:nerd_face: Message From You:

I want you to act as a regex generator. Your role is to generate regular expressions that match specific patterns in text. You should provide the regular expressions in a format that can be easily copied and pasted into a regex-enabled text editor or programming language. Do not write explanations or examples of how the regular expressions work; simply provide only the regular expressions themselves. My first prompt is to generate a regular expression that matches an email address.

:robot: Message From ChatGPT (Assistant):

Certainly! Here’s a regular expression that matches most email addresses:

\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b

Please note that this regular expression may not cover all possible valid email address formats, as the rules for email addresses can be quite complex. However, it should match the majority of commonly used email addresses.

:nerd_face: Message From You:

Now My second is prompt is to generate a regular expression that year ranged like 2021-2023.

:robot: Message From ChatGPT (Assistant):

Here’s a regular expression that matches a year range from 2021 to 2023:

\b(2021|2022|2023)\b

This regular expression will match any occurrence of the years 2021, 2022, or 2023 as standalone words. If you need to match the year range within a larger string, you can remove the word boundary \b anchors.

16

Those are exactly what you asked for. I would give it a 10 out of 10 for the prompt given.
If you interpret “spaces” as “only ASCII 32” then I would argue that you’re the one in the minority – in the Unicode world, there’s many encodings of the concept of “space” in text, and \s is the most robust encoding of that idea.

I’ve found that GPT-3.5 and GPT-4 only really understand Python and Javascript and Perl style “extended” regular expressions, though – trying to dumb it down to only be POSIX compatible regular expressions through prompting doesn’t work.

17

That was my point - that it’s responding to the prompt asked.
Maybe you replied to the wrong person?

18

Yes, the “reply to” should go to the O.P. (C123)