Here’s what I ended up implementing. I’m not 100% confident I’ve got the calculations right though:
https://tools.simonwillison.net/openai-webrtc
Source code here: tools/openai-webrtc.html at c9f3085107fd1177329846de95c840eda64b1748 · simonw/tools · GitHub