We recently discovered that when there are special symbols (<| and |>), GPT cannot see the text inside the symbols. Is this a bug in the preprocess procedure, or is this symbol intentionally designed this way? For example, consider the following prompt: Please repeat the following text: <|blocked…

You would be unlikely to experience such “text loss” in real applications. It is to prevent users from doing things like this by not having user strings encoded to special tokens of message containers or to a ValueError: [image] (tokens are + 100000) As you can see, that these can be reproduced …

Special characters causing text loss issues

Prompting

XLXW November 14, 2023, 6:44am 3

I see. Thank you for the clarification!

Topic		Replies	Views
Support of unicode in gpt4-1106-preview Bugs gpt-4 , api	10	2574	November 15, 2024
The use of the separator "-----" results in different output API gpt-35-turbo , api	13	2274	March 1, 2024
ChatGPT struggles to express the end of text token API	1	2704	February 6, 2024
Problem with : - line break interrupts text as : is used Bugs text-completion	3	163	October 24, 2024
Bug sending prompt to gtp4o using the API, not in the browser Prompting gpt-4 , api	0	183	July 12, 2024

Special characters causing text loss issues

Related topics