Failed to generate output due to special tokens in the input

API Bugs
1

I get this error very recently

openai.BadRequestError: Error code: 400 - {'error': {'message': 'Failed to generate output due to special tokens in the input.', 'type': 'invalid_request_error', 'param': None, 'code': None}}

This is a bummer because I cannot control the input! Why would I need to worry about the characters when you could tokenize anything even if that is a special character.

Furthermore, it would be helpful to get a better error message. What special character caused the issue here?

This is only a new error, was working fine few days ago

1 Like
2

I got this also when doing some scripting, sending normal example scripted stuff to embeddings, I think. Or was it functions? I’ve have to backtrack to find out what code snippet it was.

3

Since there is no actual error message, maybe it’s the opposite to the random non UTF-8 characters that we got a while back in the output?

Meaning, converting all inputs to UTF-8 (if possible for your case) could do the trick/workaround here.