Failed to generate output due to special tokens in the input

I get this error very recently

openai.BadRequestError: Error code: 400 - {'error': {'message': 'Failed to generate output due to special tokens in the input.', 'type': 'invalid_request_error', 'param': None, 'code': None}}

This is a bummer because I cannot control the input! Why would I need to worry about the characters when you could tokenize anything even if that is a special character.

Furthermore, it would be helpful to get a better error message. What special character caused the issue here?

This is only a new error, was working fine few days ago

1 Like

I got this also when doing some scripting, sending normal example scripted stuff to embeddings, I think. Or was it functions? I’ve have to backtrack to find out what code snippet it was.

Since there is no actual error message, maybe it’s the opposite to the random non UTF-8 characters that we got a while back in the output?

Meaning, converting all inputs to UTF-8 (if possible for your case) could do the trick/workaround here.