Here is a Realtime Voice API Plugin for Unreal Engine and all-talking 3D Metahumans

Community
, ,
82
  1. There is an API call that I dont think that I implemented for closing the realtime session to OpenAI
  2. This is a bit tricky as the voice data streams much faster than realtime to the client from the OpenAI audio servers. I guess you could have some sort of Blueprint trigger that is trigger on the SoundWave when it has finished playing.
1 Like
83

Sounds good, feel free to add a PR to my Fork of the OpenAI Unreal Plugin, I’ve also already made a pull request to the original creators root repo. I used o1-preview to program the Realtime additions to the plugin and I’m sure o1 or o3 mini high can help you quite a bit.

1 Like
84

You might like this :
watch?v=uUDBwUx3feo

1 Like
85

@robertb
Hey why we cannot use any other mic.
Why this plugin is working only with phone?

86

The IAudioCapture interface choose the first Mic available, I think - but you can change the code here: OpenAI-Api-Unreal/Source/OpenAIAPI/Private/OpenAIAudioCapture.cpp at main · rbjarnason/OpenAI-Api-Unreal · GitHub

87

@samirertt @robertb
I have found a solution for mic.
The problem is not with the default windows or input mic.
The main issue is that in this codes we are not converting the audio to 1 channel 44100hz.
Gpt realtime will get 1 channel 24000 or 44100 hz. All the mic that I’ve tested will Record 2 channel 44100hz.

2 Likes
89

Hello @robertb !
This plugin has been amazing so far. I have been working on implementing it all day and only need the real time api function for my uses. I’m on 5.3, using your plugin, using the marketplace realtime audio, and my blueprint looks like this: blueprintue. com/blueprint/8cypxk1e/.

My project will play and in the output log I am getting everything reading successful up until this point:

LogTemp: No create response message provided, skipping response create event
LogTemp: ===> Event Type: session.created
LogTemp: ===> Event Type: error
LogTemp: Error: Error received: Invalid event: failed to parse JSON value. Please check the value to ensure it is valid JSON. (Common errors include trailing commas, missing closing brackets, missing quotation marks, etc.)
LogTemp: WebSocket Closed: StatusCode=1000, Reason=Peer did not specify a reason for initiating the closing, WasClean=1

Nothing ever crashes, but nothing ever happens either. I’ll speak into the mic (tried several different microphones with both USB and 3.5mm jack connections) with no successful audio capture. I’ve also got my API key loaded and money loaded on the OpenAI account. Do you have any ideas?

Thank you so much!

1 Like
90

@nrekow2 Are there any clues further up in the log file? Looks like the connection to OpenAI is not working; is the API key being transferred and setup correctly? The JSON failure could be due a null value coming in through the WebSockets.

91

Just as an update and to anyone else who had the issue that I did: I traced my issue down to a ’ in the instructions that was not being decoded correctly and was being sent as a \ instead, causing a JSON parsing error. Everything works now and a big big thank you to @robertb for this amazing plugin :slight_smile:

1 Like