Why do I sometimes receive a null response when using GPT-4 with pre-instructions, even though it works correctly in the playground with the same pre-instruction? Can you help me understand this?
Do you have your API call wrapped in a try/except error catcher? It could be that there is some issue occurring that is causing an error and you are missing it.
Can you share the code you are using ?
The API response is returned in a specific key in the return dictionary, maybe you are catching it wrongly/
1 Like
As you said, I have wrapped the API call code with try/catch, but no error is displayed whenever I receive a null response.
A null response can be the AI simply having nothing to write. It sends a special token that indicates it is done writing.
This would be quite unusual on gpt-4, given its training, but is easy to show on other models such as text-davinci-003.
Here is a preset I’ve created. Press submit. You don’t get anything back except for a message that the AI generated a stop token.
For gpt-4, what could have gone wrong? You’ll need to ensure you are giving it the expected input format. “pre-instructions” is a strange phrasing; rather is a particular format we’d want to use:
system role message: AI name, task, purpose, behaviors, formatting
user role message: instruction or data
for example:
system:
You are ChatGPT. You are a friendly chatbot.
New behavior: Mention fruits a lot in every reply, just to be silly.
Banana is a particularly funny fruit.
user:
Compose a quick note to my mom about my pet cat Spot.
So you can look at your inputs. If you have a user role, don’t just plop data there for processing, as the data may be seen as “done”. Add a little instruction prefix such as “Here is data to process: {my_data}”
1 Like
Is it possible that the token size of the pre-instruction is causing this issue? My current instruction consists of 3k tokens, while GPT-4 supports up to 8k tokens. I previously completed another project with 2k tokens, and it performed well without any null responses.
As long as you are using max_tokens, you’ll have that space available for formation of an answer. Send or reserve too much data, and the reservation for output won’t get smaller, you’ll just get an error.
API Error: This model’s maximum context length is 4097 tokens. However, you requested 9117 tokens (117 in the messages, 9000 in the completion). Please reduce the length of the messages or completion.
You of course must perform error handling to see that error and not just ignore it, and should be logging the full response to see if you get the normal response container with empty assistant message as contents
"message": {
"role": "assistant",
"content": "I reply here"
The daily usage will also show if you are getting billed for 1 or 0 response tokens by those calls, and it is not some problem with your handling of the output.