I am using the C# Assistants API. I have observed that sometimes a chat response run is automatically interrupted, throwing OperationCanceledException. I do not yet know under what conditions this occurs, and I cannot reliably reproduce it.
I pass a CancellationToken to the API calls that accept it as a parameter. My expectation is that such an operation will only throw OperationCanceledException if the token I submit is cancelled (unless otherwise clearly documented). The token that is being submitted is not being cancelled in these cases.
I suspect that the CancellationToken might be thrown due to some type of timeout. However, I expect a TimeoutException for such cases – not OperationCanceledException.
This behavior is confusing to me, and it is making it difficult to program against the API in a consistent and reliable manner. Can anyone explain to me what is causing this?
I encountered a similar error that was difficult to reproduce. After some effort, I was able to resolve the issue by explicitly passing a Timeout parameter to runs.stream(). I suspect this is related to the default timeout value in httpx, which is set to 5 seconds. Sometimes, 5 seconds isn’t enough, especially when the OpenAI service is experiencing heavy traffic. I’m using Python, and here’s an example of how I fixed it:
async with client.beta.threads.runs.stream(
thread_id=thread_id,
assistant_id=openai_assistant_id,
tools=self.tool_calls,
timeout=Timeout(10.0),
) as stream:
Hope this helps!
Thanks, @bookholic.tw. I appreciate your attempt to help by sharing your similar experiences.
After further debugging I have confirmed that the cause of the OperationCanceledException is a timeout limit being exceeded. This happens when the assistant fails to send a thread run update within 100 seconds.
I do consider this to be a ‘bug’, or at the least an undocumented and ambiguous output value from the API operation.
With C#, when an operation accepts a CancellationToken, the standard behavior is for the operation to throw OperationCanceledException if AND ONLY IF that CancellationToken is triggered. That allows callers to catch OperationCanceledException and handle it consistently. If I pass a CancellationToken that is triggered due to a timeout being exceeded, then OperationCanceledException should be thrown. If the operation happens to use its own CancellationToken that is triggered due to a timeout being exceeded, that is an implementation detail that should NOT cause OperationCanceledException to be thrown. Instead, the exception should be caught internally, and an appropriate result should be returned or some kind of “Timeout” exception should be thrown.
As a workaround, I am now catching OperationCanceledException and then checking to see if my CancellationToken has been triggered. If it has been, I rethrow the original exception. If not, I wrap the OperationCanceledException in an exception of another type and throw it.