I had the same issue. It turns out the culprit was input audio noise reduction:
"input_audio_noise_reduction": {
"type": "far_field"
}
I removed it and the connection worked just fine. I suspect this is a bug on OpenAI side, given that 1) the noise reduction is a documented feature of the create session endoint and 2) it worked fine last week.