ChatGPT fails when identifying bits in a hexadecimal value

ChatGPT Bugs
1

Hi I’m trying to use ChatGPT 3.5-turbo model to identify faults, status and other values into a register. For example, I provide the IA with a context about the register definition (hipotetical register from my invention). The context contains the register definition, what is used for, the address, the lenght and all the bit fields definition. I indicate that is bigendian format and everything. But when I ask the IA to analyze a value and try to identify the meaning bit by bit it not just fails when mapping the bits with my register definition but it also fails in a basic tasks like saying that a bit is clear or set when it is not.
For example:
prompt: The value in hexadecimal of the 16bit length “register” is 0x0200. Identify bit by bit what faults are caused and the meaning of each one set.

Response: Sure, in the value 0x0200 which is 0b0000010000000000 in binary the bit 0, bit 1 and bit 8 are set which means: bla bla bla (if those bits were set, the answer is correct).

So if I continue asking for different values sometimes is rigth and sometimes is wrong (most of times is wrong). I tried to give it some examples, tell it what is the bit 0, how it is incremented, that more than one bit could be set and always fails. What concerns to me is that even if the values conversion are good the IA is not able to identify if a bit is set or clear even when the conversion and the bits idenfitication is done right. That makes the analysis to crash.
The only way it does not fails is giving an example of all the possible values in the register (I tried this in a 4 bits register), but imagine the mess for a 16bit or 32bit register.

Does anybody has detected this issue, is this a bug or this is something beyond the capabilities of the ChatGPT (which is weird cause is just simply binary aritmetic)? Maybe I’m wrong with the way I’m giving it context, the format presented (just text and descriptions) or the way I’m doing the prompts. Hope anybody can help me, this is hanging me in the investigation I’m doing for my thesis :frowning:

2

Yes ChatGPT, using OpenAI’s latest model, fails stupendously on the simple.

Sure, I can help you with that. Let’s break it down step by step.

First, let’s display each hex character as four bits in a table with four rows:

---------------------------------
| 0 | 2 | 0 | 0 |
---------------------------------

Next, let’s display each bit by position in the value in a table with sixteen rows, with columns for bit number, value, and purpose:

------------------------------------------------
| Bit Number | Value | Purpose                |
------------------------------------------------
|    15      |   0   | Address Bit            |
|    14      |   0   | Address Bit            |
|    13      |   0   | Address Bit            |
|    12      |   0   | Address Bit            |
|    11      |   0   | Address Bit            |
|    10      |   0   | Address Bit            |
|     9      |   0   | Address Bit            |
|     8      |   0   | Address Bit            |
|     7      |   0   | Address Bit            |
|     6      |   0   | Address Bit            |
|     5      |   0   | Address Bit            |
|     4      |   0   | Address Bit            |
|     3      |   0   | Length Value Bit       |
|     2      |   0   | Length Value Bit       |
|     1      |   0   | Length Value Bit       |
|     0      |   0   | Error Condition Bit    |
------------------------------------------------

Finally, let’s display a summary with the calculated address, calculated length, and error value bits that are set:

  • Calculated Address: 0x0000 (In hexadecimal)
  • Calculated Length: 0x000 (In hexadecimal)
  • Error Value Bits: 000 (In binary)

I hope this information helps! If you have any further questions or need additional assistance, please let me know.

Give it to API gpt-3.5-turbo-0301 from last March, at top_p:0.1, though, we get some bits set, and much better presentation, but “address” not translated into the bits or bytes of an address.

Here are the tables and summary for the given register:

Table 1: Hexadecimal Value in Four-Bit Rows

15-12 11-8 7-4 3-0
0 2 0 0

Table 2: Bit Positions and Values

Bit Number Value Purpose
15 0 Address Bit
14 0 Address Bit
13 1 Address Bit
12 0 Address Bit
11 0 Address Bit
10 0 Address Bit
9 0 Address Bit
8 1 Address Bit
7 0 Length Value Bit
6 0 Length Value Bit
5 0 Length Value Bit
4 0 Error Condition Bit 2
3 0 Error Condition Bit 1
2 0 Error Condition Bit 0

Summary:

Calculated Address: 0x0200
Calculated Length: 0x00
Error Value Bits: 0x00 (No error condition bits are set)

Prompt I had to imagine based on what you are talking about. You can see I tried to segment the tasks.

Prompt

register definition:
15-6: address bits
5-3: length value bits
2-0: bits indicate error condition 2, 1, 0

The value in hexadecimal of a MSB 16 bit register that was received is 0x0200.

Your tasks, outputting a table for each:

  • display each hex character as four bits in a table with four rows
  • display each bit by position in value in a table with sixteen rows, with columns for bit number, value, purpose
  • display a summary with calculated address, calculated length, and error value bits that are set.

The presentation of information shall make clear to the user all information available.

3

Thank you so much for your help. I will try the model you suggested but as far I understood it still fail. Is there a way to report this like a bug or something to be fixed? If this simple task cannot be performed by the model, then I’m fried! Do you know if the GPT 4 works better? I know is in beta but if it works I don’t mind paying for the early access

4

GPT-4 is in wide release, a version is within ChatGPT plus, and multiple model version available on API for paying API developers.

The closest to coherency is GPT-4-0314. The prompt is still lacking actual application, but everything good until we try to extract an address from high MSB.

User:

register definition:
15-6: address bits
5-3: length value bits
2-0: bits indicate error condition 2, 1, 0

The value in hexadecimal of a MSB 16 bit register that was received is 0x0200.

Your tasks, outputting a table for each:

  • display each hex character as four bits in a table with four rows, labeled 15-12, …
  • display each bit by position in value in a table with sixteen rows, with columns for bit number, value, purpose
  • determine how many bits are in just the address part of the register
  • extract just the address bits, by the smaller address space and new bit values end at 0 at bit 6 of input.
  • display a summary with calculated address (the output value of the address bits), calculated length, and error value bits that are set.

The presentation of information shall make clear to the user all information available.

Output:

Task 1: Display each hex character as four bits in a table with four rows, labeled 15-12, …

15-12 11-8 7-4 3-0
0000 0010 0000 0000

Task 2: Display each bit by position in value in a table with sixteen rows, with columns for bit number, value, purpose

Bit Number Value Purpose
15 0 Address Bit
14 0 Address Bit
13 0 Address Bit
12 0 Address Bit
11 0 Address Bit
10 1 Address Bit
9 0 Address Bit
8 0 Address Bit
7 0 Address Bit
6 0 Address Bit
5 0 Length Value Bit
4 0 Length Value Bit
3 0 Length Value Bit
2 0 Error Bit 2
1 0 Error Bit 1
0 0 Error Bit 0

Task 3: Determine how many bits are in just the address part of the register

There are 10 bits in the address part of the register (bits 15-6).

Task 4: Extract just the address bits, by the smaller address space and new bit values end at 0 at bit 6 of input.

9-6 5-2 1-0
0100 0000 00

Task 5: Display a summary with calculated address (the output value of the address bits), calculated length, and error value bits that are set.

Calculated Address Calculated Length Error Value Bits
0x4 0 0

Summary:

  • The calculated address is 0x4 (4 in decimal).
  • The calculated length is 0.
  • There are no error value bits set

Again, if you thought up a real application and the results you wanted to get out, and also gave the AI a system message that made it a specialist, it could probably do some stuff for you.

In general though, I’ve used prompt tricks to make the AI pay attention to the bits and the locations by the type of multi-step output it produces. It is not very good at such “math”, as it is designed to generate language.