Same result for me as well. I already have a pile of code written… just waiting for access lol
2 Likes
dang, I haven’t even started to write code for it, I’m not a huge fan of writing software blindly. so, what did you write so far for it?
I have a project I did a few weeks back that is a self modifying UI. It’s a web server that returns pages that are 100% AI generated. Each page has a chat feed and when you type in an instruction that instruction plus the page is sent to the model and the page is completely re-written. You can tell it create a Tetris game and it will complete re-write the page to include a Tetris game.
I’m updating that project to support voice. It will basically be the Star Trek computer. You can tell it what kind of app you want or what you want it to display and it will re-write the current page.
The old code did everything with one model and in the new code I’m splitting it such that realtime handles the chat and Claude 3.5 sonnet does all the coding.
I was ready to re-write it all anyway so this is a good chance to redesign everything.
I renamed the project as well… it was called ShellE but now it’s called SynthOS.
I have an idea for how to implement code that’s self healing so I’m hoping to work that into the new architecture as well.
2 Likes
nice, sounds like fun. I’ve done something similar, my own “devin” or what replit is doing now. I’ll probably create a text based assistant first once it is out.
Did you write your code based off the openai-realtime-api-beta repository?
1 Like
any idea when access is meant to be rolled out?
I just pasted in the code from their console app so I can do a basic test. I’m mostly focused on splitting the model calls. I’ll use tool calls to bridge the gap between models but I need to see it actually running to get a feel for the instructions and flow
“throughout the week”
I don’t have access yet
One thing I can already tell you is that given the pricing you’re going to need to pair realtime with another model. It’s that pairing I want to work out.
It’s actually kind of serendipitous because with my SynthOS project I was already coming to the conclusion that the conversational aspects of the UX should be separated from the code generation piece. The correct architecture for SynthOS is really just a bunch of different Micro Agents handling different jobs.
I’m mostly waiting to see what they are going to do with resampling the PCM data. there’s an example client at the openai-realtime-console repository.
but since the modal is not out, I don’t know if there is going to be changes or what.
I saw a PR today that if I remember correctly addressed a resampling issue in Firefox
1 Like
I want a better understanding of tool execution pricing. Are tools billed as text tokens or audio tokens?
From an OpenAI Employee I messaged:
RealTime API is being released in phases so you should gain access over the next day or two! Looking forward to hearing what you build when you get your hands on it!
Just a waiting game now, I guess it makes sense as otherwise everyone would start a session at once!
4 Likes
oh, that was me, i wrote that pr
oh thank god @trenton.dambrowitz, thank you for this, really appreciate it
1 Like
Hello, I am on this page after trying to update a websocket system that I already had set up. I was transporting the audio buffer of an interaction with the user with the corresponding stt(whisper) → ai → tts(tts-1). I wanted to try this new realtime API. Since I already had this system set up (with authorization headers), it was easy.
But when I tried it, I found “The HTTP response from the server [403] did not permit the HTTP upgrade to WebSocket”.
What is the expected ETA on this for the rest of us that still don’t have access to it? Are we talking days or weeks? Would be nice to get some updates from OpenAI, the suspense is killing slowly from the inside out!
I get that, but I’d rather have a word from OpenAI saying that it will take a month until I have access, instead of just sitting here like a child waiting for Christmas Eve. Actually, that’s not a good analogy since you know exactly when Christmas Eve is 
2 Likes
Server is listening on port 5050
node:events:497
throw er; // Unhandled ‘error’ event
^
Error: Unexpected server response: 403
at ClientRequest. (D:\WORKSPACE\DUMPSTER_RENTAL\dumpster-rental\node_modules\ws\lib\websocket.js:913:7)
at ClientRequest.emit (node:events:519:28)
Dose anyone have idea about on this error
@elm just saw some footage of the fireside chat with sam, first question was “how close are we to AGI?”. really cool to see it started like this! hopefully I’ll be able to go to next year’s dev day, fingers crossed