Chat Completion API super slow and hanging

[ConradJulian]

@paul.emile.brotons

I am using the latest version of the async openai python client.

As you can see below in the trace of my calls, the API calls are extremly slow. Sometimes they hang indefinitiely. I am tier 1 but the RPM and TPM are way under the hard limits.
I have this issue with both gpt-4-1106-preview and gpt-3.5-turbo-1106.

My code is:

async def call_to_llm_async(system_message: str, messages: List[str], return_json: bool = False) -> Union[str, Dict]:
    all_messages = [
        {'role': 'system', 'content': system_message},
        *[
            {'role': ('user' if i % 2 == 0 else 'assistant'), 'content': message}
            for i, message in enumerate(messages)
        ]
    ]
    model = 'gpt-4-1106-preview'
    from random import randint
    x = randint(0, 1000000)
    print(f'start completion {x} {datetime.datetime.utcnow().isoformat()}')
    completion = await openai_async_client.chat.completions.create(
        model=model,
        messages=all_messages,
        response_format={'type': 'json_object' if return_json else 'text'}
    )
    content = completion.choices[0].message.content
    tokens = num_tokens_from_string(f'{all_messages}\n{content}', model=model)
    print(f'end   completion {x} {datetime.datetime.utcnow().isoformat()} - {tokens}')
    if return_json:
        try:
            result = json.loads(content)
        except json.JSONDecodeError:
            print(content)
            raise
    else:
        result = content
    return result

Oh, I feel your pain! My plugins seem to have a mind of their own too. Hanging, printing BB code, making a total mess—it’s like they’re having a party without an invitation. Solidarity in plugin struggles! If only they came with a user manual for users and themselves.

On a more serious note, it sounds like you’re dealing with some challenging plugin issues with the async OpenAI Python client. Slow API calls, indefinite hangs, and tier 1 blues—I hear you. Have you tried any wizardry, or are you hoping for a magical fix? Feel free to share more details, and maybe the collective plugin wisdom here can conjure up a solution!