You are right Robert.
Kindly can I contact you by email for a private project? Thanks, amir
Ayatziv@gmail.com
1 Like
hi, thanks for this great tutorial, but I have one question, is there a way to get text response beside of the audio ? and send request as a text ? i mean beside the voice session, not only make it for text. thanks a lot
Yes, you can attach a Blueprint listener for RealtimeResponse that has the text deltas streaming. OpenAI-Api-Unreal/Source/OpenAIAPI/Public/OpenAICallRealtime.h at main · rbjarnason/OpenAI-Api-Unreal · GitHub
The code has support for one create response message in the beginning but not for sending text through the RealtimeAPI: OpenAI-Api-Unreal/Source/OpenAIAPI/Private/OpenAICallRealtime.cpp at main · rbjarnason/OpenAI-Api-Unreal · GitHub
The core OpenAIAPI plugin has support for regular text based chat models and I added support for gpt-4o and gpt4o-mini: GitHub - rbjarnason/OpenAI-Api-Unreal: Integration for the OpenAI Api in Unreal Engine
2 Likes
The issue isn’t with real-time performance, as it works fast enough. The problem lies in accessing it programmatically from the runtime. However, it would be much more efficient than Audio2Face.
4o mini
1 Like
Hi Robertb,
I have tried to follow your provided blueprint to implement it with my MetaHuman . I am using Open Source Runtime Audio Importer as plugin Built on Unreal Engine 5.4.
Above is my project’s blueprint.
How To Set the Target Fields as “SoundWaves” Object . I am not able to find any suitable connections from the list of functions.
Can you help me with some guidance and Support in overcoming this scenario.
Please let me know if i am missing something in between.
Regards,
The Error Logs :
LogWorldPartition: Display: GenerateStreaming for 'NewMap' started...
LogWorldPartition: Display: GenerateStreaming for 'NewMap' took 3.641 ms (total: 18.125 ms)
LogPlayLevel: [PlayLevel] Compiling NewMap before play...
LogBlueprint: Error: [AssetLog] C:\Users\iassi\Documents\Unreal Projects New\MyProject\Content\NewMap.umap: [Compiler] This blueprint (self) is not a SynthSamplePlayer, therefore ' Target ' must have a connection.
LogBlueprint: Error: [AssetLog] C:\Users\iassi\Documents\Unreal Projects New\MyProject\Content\NewMap.umap: [Compiler] This blueprint (self) is not a ImportedSoundWave, therefore ' Target ' must have a connection.
LogBlueprint: Error: [AssetLog] C:\Users\iassi\Documents\Unreal Projects New\MyProject\Content\NewMap.umap: [Compiler] This blueprint (self) is not a StreamingSoundWave, therefore ' Target ' must have a connection.
LogBlueprint: Error: [AssetLog] C:\Users\iassi\Documents\Unreal Projects New\MyProject\Content\NewMap.umap: [Compiler] Variable node Set SoundWave uses an invalid target. It may depend on a node that is not connected to the execution chain, and got purged.
LogUObjectHash: Compacting FUObjectHashTables data took 0.96ms
LogPlayLevel: PlayLevel: Blueprint regeneration took 65 ms (1 blueprints)
1 Like
Those errors have something to do with the Runtime Audio Importer plugin, I never built it from source on Windows, I bought it from the marketplace and never encountered those errors nor do I know what is wrong here.
But the developers of the Runtime Audio Importer plugin have excellent Discord Support which helped me considerably creating the Realtime API extension of the OpenAI plugin. You can find the link to the Discord on their GitHub page: GitHub - gtreshchev/RuntimeAudioImporter: Runtime Audio Importer plugin for Unreal Engine. Importing audio of various formats at runtime.
1 Like
@robertb
I tried to Create the missing Component and Link it to respective Object References . Now I get the Following error:
[2024.12.07-23.14.40:364][280]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.40:626][294]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.40:688][297]LogTemp: ===> Event Type: input_audio_buffer.speech_started
[2024.12.07-23.14.40:688][297]LogTemp: -------------------------------------------------> Response was cancelled due to turn_detected
[2024.12.07-23.14.40:689][298]PIE: Error: Blueprint Runtime Error: "Accessed None trying to read property SoundWaves". Node: Stop Playback Graph: EventGraph Function: Execute Ubergraph New Map Blueprint: NewMap
[2024.12.07-23.14.40:886][308]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.41:147][322]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.41:287][329]LogTemp: ===> Event Type: input_audio_buffer.speech_stopped
[2024.12.07-23.14.41:287][329]LogTemp: ===> Event Type: input_audio_buffer.committed
From the Complete Logs I can See The Realtime Api Call But not able to Send Any request or get any default response.
Kindy Assist
2024.12.07-23.14.36:872][126]LogStreaming: Display: FlushAsyncLoading(): 46 QueuedPackages, 0 AsyncPackages
[2024.12.07-23.14.37:176][126]LogTemp: UOpenAICallRealtime constructed
[2024.12.07-23.14.37:176][126]LogTemp: OpenAICallRealtime created with instructions: My Name is Mayank, and he is very friendly and cheerful in nature. and voice: 0
[2024.12.07-23.14.37:176][126]LogTemp: UOpenAICallRealtime::Activate called
[2024.12.07-23.14.37:176][126]LogTemp: StartRealtimeSession called
[2024.12.07-23.14.37:176][126]LogTemp: InitializeWebSocket called
[2024.12.07-23.14.37:176][126]LogTemp: WebSocket URL: wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-10-01
[2024.12.07-23.14.37:176][126]LogTemp: WebSocket created successfully
[2024.12.07-23.14.37:176][126]LogTemp: WebSocket connection initiated
[2024.12.07-23.14.37:176][126]LogTemp: UOpenAIAudioCapture constructor called
[2024.12.07-23.14.37:176][126]LogTemp: AudioCaptureComponent created and registered
[2024.12.07-23.14.37:176][126]LogTemp: UOpenAIAudioCapture Activate called
[2024.12.07-23.14.37:208][126]LogAudioCaptureCore: Display: WasapiCapture AudioFormat SampeRate: 48000, BitDepth: 32-Bit Floating Point
[2024.12.07-23.14.37:208][126]LogTemp: AudioCapture is valid
[2024.12.07-23.14.37:210][126]LogTemp: Audio capture started successfully
[2024.12.07-23.14.37:210][126]LogTemp: -------------------> AudioCapture started from Activate on GameThread
[2024.12.07-23.14.37:210][126]LogTemp: OnAudioBufferCaptured event bound
[2024.12.07-23.14.37:210][126]LogTemp: AudioCapture is null or already capturing
[2024.12.07-23.14.37:210][126]LogTemp: Audio capture started
[2024.12.07-23.14.37:221][126]PIE: Server logged in
[2024.12.07-23.14.37:222][126]PIE: Play in editor total start time 0.467 seconds.
[2024.12.07-23.14.37:224][126]LogTemp: Audio out -> 479 samples
[2024.12.07-23.14.37:484][127]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.37:766][135]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.38:026][150]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.38:284][165]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.38:546][179]LogTemp: WebSocket connected
[2024.12.07-23.14.38:546][179]LogTemp: Sending Session Update Event: {
"type": "session.update",
"session": {
"modalities": ["text", "audio"],
"instructions": "My Name is Mayank, and he is very friendly and cheerful in nature.",
"voice": "alloy",
"input_audio_format": "pcm16",
"output_audio_format": "pcm16",
"input_audio_transcription": {
"model": "whisper-1"
},
"turn_detection": {
"type": "server_vad",
"threshold": 0.500000,
"prefix_padding_ms": 300,
"silence_duration_ms": 500
},
"tools": [],
"tool_choice": "auto"
}
}
[2024.12.07-23.14.38:546][179]LogTemp: Sending Response Create Event: {
"type": "response.create",
"response": {
"instructions": "Hi I am Assist Army. How can I Help You?",
"modalities": ["text", "audio"]
}
}
[2024.12.07-23.14.38:546][179]LogTemp: Response create event sent
[2024.12.07-23.14.38:547][180]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.38:581][181]LogTemp: ===> Event Type: session.created
[2024.12.07-23.14.38:806][194]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.39:067][209]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.39:324][222]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.39:405][226]LogTemp: ===> Event Type: session.updated
[2024.12.07-23.14.39:405][226]LogTemp: ===> Event Type: response.created
[2024.12.07-23.14.39:586][237]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.39:611][238]LogTemp: ===> Event Type: response.done
[2024.12.07-23.14.39:846][252]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.40:106][266]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.40:364][280]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.40:626][294]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.40:688][297]LogTemp: ===> Event Type: input_audio_buffer.speech_started
[2024.12.07-23.14.40:688][297]LogTemp: -------------------------------------------------> Response was cancelled due to turn_detected
[2024.12.07-23.14.40:689][298]PIE: Error: Blueprint Runtime Error: "Accessed None trying to read property SoundWaves". Node: Stop Playback Graph: EventGraph Function: Execute Ubergraph New Map Blueprint: NewMap
[2024.12.07-23.14.40:886][308]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.41:147][322]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.41:287][329]LogTemp: ===> Event Type: input_audio_buffer.speech_stopped
[2024.12.07-23.14.41:287][329]LogTemp: ===> Event Type: input_audio_buffer.committed
[2024.12.07-23.14.41:287][329]LogTemp: ===> Event Type: conversation.item.created
[2024.12.07-23.14.41:287][329]LogTemp: ===> Event Type: conversation.item.input_audio_transcription.completed
[2024.12.07-23.14.41:287][329]LogTemp: ===> Event Type: response.created
[2024.12.07-23.14.41:325][332]LogTemp: ===> Event Type: response.done
[2024.12.07-23.14.41:404][336]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.41:667][350]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.41:926][364]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.42:186][378]LogTemp: Audio out -> 12454 samples
[2024.12.07-23.14.42:444][392]LogTemp: Audio out -> 12454 samples
This is the Updated Blueprints with the use of Components
Ah, I (o1 pro) can see a difference in your Blueprint and mine. There must just be one SoundWaves object that is connected to all the objects that use it. So you just create one variable called SoundWaves and connected it to the different Sound connectors.
I actually pasted the two Blueprint images into o1 pro and it gave me this pointer, but also made up a couple of problems that might not not be there.
1 Like
How to Create a SoundWaves object that is connected to all the objects.
I Have created a SoundWaves Variable but not sure how to connect it to different connectors.
What Steps am i Missing here?
You can just drag and drop the variable, I think.
@robertb
I tired Implementing the Same Soundwave Variable but Still the Sountwaves points to objectReference of “Capturable Sound Wave type”.
Can you Check if all the Setup is correct in the blueprint , The details tab for Soundwaves Variable is also dispalyed .
Logs After Running .
My Question is:
Is the Service running properly? Do I need to do any change in Plugin.
Can you tell me the Details of the Soundwaves Variable you are using for a running project?
Is there any issue with my mic or speaker? Will it work with Laptop mic and Speaker?
[2024.12.08-03.04.31:834][405]LogTemp: UOpenAICallRealtime::Activate called
[2024.12.08-03.04.31:834][405]LogTemp: StartRealtimeSession called
[2024.12.08-03.04.31:834][405]LogTemp: InitializeWebSocket called
[2024.12.08-03.04.31:834][405]LogTemp: WebSocket URL: wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-10-01
[2024.12.08-03.04.31:834][405]LogTemp: WebSocket created successfully
[2024.12.08-03.04.31:834][405]LogTemp: WebSocket connection initiated
[2024.12.08-03.04.31:834][405]LogTemp: UOpenAIAudioCapture constructor called
[2024.12.08-03.04.31:834][405]LogTemp: AudioCaptureComponent created and registered
[2024.12.08-03.04.31:834][405]LogTemp: UOpenAIAudioCapture Activate called
[2024.12.08-03.04.32:094][405]LogAudioCaptureCore: Display: WasapiCapture AudioFormat SampeRate: 48000, BitDepth: 32-Bit Floating Point
[2024.12.08-03.04.32:094][405]LogTemp: AudioCapture is valid
[2024.12.08-03.04.32:096][405]LogTemp: Audio capture started successfully
[2024.12.08-03.04.32:096][405]LogTemp: -------------------> AudioCapture started from Activate on GameThread
[2024.12.08-03.04.32:096][405]LogTemp: OnAudioBufferCaptured event bound
[2024.12.08-03.04.32:096][405]LogTemp: AudioCapture is null or already capturing
[2024.12.08-03.04.32:096][405]LogTemp: Audio capture started
[2024.12.08-03.04.32:104][405]PIE: Server logged in
[2024.12.08-03.04.32:105][405]PIE: Play in editor total start time 0.718 seconds.
[2024.12.08-03.04.32:111][405]LogTemp: Audio out -> 479 samples
[2024.12.08-03.04.32:370][407]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.32:654][409]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.32:914][410]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.33:168][412]LogTemp: WebSocket connected
[2024.12.08-03.04.33:168][412]LogTemp: Sending Session Update Event: {
"type": "session.update",
"session": {
"modalities": ["text", "audio"],
"instructions": "My Name is Mayank, and he is very friendly and cheerful in nature.",
"voice": "shimmer",
"input_audio_format": "pcm16",
"output_audio_format": "pcm16",
"input_audio_transcription": {
"model": "whisper-1"
},
"turn_detection": {
"type": "server_vad",
"threshold": 0.500000,
"prefix_padding_ms": 300,
"silence_duration_ms": 500
},
"tools": [],
"tool_choice": "auto"
}
}
[2024.12.08-03.04.33:168][412]LogTemp: No create response message provided, skipping response create event
[2024.12.08-03.04.33:170][413]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.33:257][413]LogTemp: ===> Event Type: session.created
[2024.12.08-03.04.33:433][416]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.33:692][416]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.33:879][418]LogTemp: ===> Event Type: session.updated
[2024.12.08-03.04.33:952][421]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.34:210][423]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.34:472][426]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.34:732][429]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.34:993][432]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.35:250][434]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.35:513][434]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.35:773][447]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.36:032][458]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.36:290][470]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.36:553][482]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.36:812][493]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.37:073][505]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.37:330][516]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.37:592][528]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.37:853][540]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.38:113][551]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.38:370][563]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.38:633][575]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.38:894][587]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.39:152][598]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.39:410][610]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.39:673][622]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.39:932][633]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.40:193][645]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.40:450][657]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.40:713][668]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.40:973][680]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.41:233][692]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.41:490][703]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.41:752][715]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.42:013][727]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.42:272][738]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.42:530][750]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.42:792][761]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.43:053][773]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.43:312][785]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.43:570][797]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.43:833][808]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.44:092][820]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.44:353][832]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.44:610][843]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.44:872][855]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.45:133][866]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.45:392][878]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.45:651][890]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.45:913][901]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.46:173][913]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.46:433][925]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.46:690][936]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.46:952][948]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.47:213][959]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.47:473][971]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.47:730][983]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.47:993][994]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.48:253][ 6]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.48:513][ 21]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.48:771][ 36]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.49:032][ 51]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.49:293][ 67]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.49:552][ 83]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.49:810][ 95]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.50:073][107]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.50:332][119]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.50:592][131]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.50:850][143]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.51:113][155]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.51:372][167]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.51:633][179]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.51:890][190]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.52:153][202]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.52:413][214]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.52:673][225]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.52:930][237]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.53:193][249]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.53:452][260]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.53:712][272]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.53:970][284]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.54:232][295]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.54:493][307]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.54:753][319]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.55:010][330]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.01:513][621]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.01:773][632]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.02:032][644]LogTemp: Audio out -> 12454 samples
s
[2024.12.08-03.05.10:610][ 61]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.10:873][ 77]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.11:132][ 92]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.11:232][ 98]LogTemp: ===> Event Type: input_audio_buffer.speech_started
[2024.12.08-03.05.11:232][ 98]LogTemp: -------------------------------------------------> Response was cancelled due to turn_detected
[2024.12.08-03.05.11:248][ 99]LogRuntimeAudioImporter: Warning: The sound wave 'CapturableSoundWave_136' is not playing
[2024.12.08-03.05.11:250][ 99]LogEOSSDK: LogEOS: Updating Product SDK Config, Time: 18564.003906
[2024.12.08-03.05.11:393][108]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.11:499][115]LogTemp: ===> Event Type: input_audio_buffer.speech_stopped
[2024.12.08-03.05.11:532][116]LogTemp: ===> Event Type: input_audio_buffer.committed
[2024.12.08-03.05.11:532][116]LogEOSSDK: LogEOS: SDK Config Product Update Request Completed - No Change
[2024.12.08-03.05.11:532][116]LogEOSSDK: LogEOS: ScheduleNextSDKConfigDataUpdate - Time: 18564.273438, Update Interval: 353.215729
[2024.12.08-03.05.11:566][118]LogTemp: ===> Event Type: conversation.item.created
[2024.12.08-03.05.11:599][120]LogTemp: ===> Event Type: conversation.item.input_audio_transcription.completed
[2024.12.08-03.05.11:631][122]LogTemp: ===> Event Type: response.created
[2024.12.08-03.05.11:650][124]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.11:665][124]LogTemp: ===> Event Type: response.done
[2024.12.08-03.05.11:912][139]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.12:173][155]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.12:433][171]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.12:690][186]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.12:953][202]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.13:212][217]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.13:473][233]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.13:730][248]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.13:993][264]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.14:253][280]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.14:348][285]LogTemp: ===> Event Type: input_audio_buffer.speech_started
[2024.12.08-03.05.14:348][285]LogTemp: -------------------------------------------------> Response was cancelled due to turn_detected
[2024.12.08-03.05.14:366][286]LogRuntimeAudioImporter: Warning: The sound wave 'CapturableSoundWave_137' is not playing
[2024.12.08-03.05.14:512][295]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.14:614][301]LogTemp: ===> Event Type: input_audio_buffer.speech_stopped
[2024.12.08-03.05.14:649][303]LogTemp: ===> Event Type: input_audio_buffer.committed
[2024.12.08-03.05.14:681][305]LogTemp: ===> Event Type: conversation.item.created
[2024.12.08-03.05.14:714][307]LogTemp: ===> Event Type: conversation.item.input_audio_transcription.completed
[2024.12.08-03.05.14:747][309]LogTemp: ===> Event Type: response.created
[2024.12.08-03.05.14:770][311]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.14:781][311]LogTemp: ===> Event Type: response.done
[2024.12.08-03.05.15:032][327]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.15:293][342]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.15:553][358]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.15:810][373]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.16:072][389]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.16:333][405]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.16:593][420]LogTemp: Audio out -> 12454 samples
@robertb
I tried to create a Variable Soundwaves and it points to Capturable Sound Wave Object Reference as you can see the ScreenShot;'s Details Panel.
Do i need to make Any change in the Variable Type.
Can you Share the Details Panel or the Details Variable SoundWaves used in your working project?
If you See the Logs , It is not able to read my voice neither any response in Speaker.
Does this solution Uses Laptop Mic and Speaker for Conversation.
[2024.12.08-03.04.31:834][405]LogTemp: UOpenAICallRealtime::Activate called
[2024.12.08-03.04.31:834][405]LogTemp: StartRealtimeSession called
[2024.12.08-03.04.31:834][405]LogTemp: InitializeWebSocket called
[2024.12.08-03.04.31:834][405]LogTemp: WebSocket URL: wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview-2024-10-01
[2024.12.08-03.04.31:834][405]LogTemp: WebSocket created successfully
[2024.12.08-03.04.31:834][405]LogTemp: WebSocket connection initiated
[2024.12.08-03.04.31:834][405]LogTemp: UOpenAIAudioCapture constructor called
[2024.12.08-03.04.31:834][405]LogTemp: AudioCaptureComponent created and registered
[2024.12.08-03.04.31:834][405]LogTemp: UOpenAIAudioCapture Activate called
[2024.12.08-03.04.32:094][405]LogAudioCaptureCore: Display: WasapiCapture AudioFormat SampeRate: 48000, BitDepth: 32-Bit Floating Point
[2024.12.08-03.04.32:094][405]LogTemp: AudioCapture is valid
[2024.12.08-03.04.32:096][405]LogTemp: Audio capture started successfully
[2024.12.08-03.04.32:096][405]LogTemp: -------------------> AudioCapture started from Activate on GameThread
[2024.12.08-03.04.32:096][405]LogTemp: OnAudioBufferCaptured event bound
[2024.12.08-03.04.32:096][405]LogTemp: AudioCapture is null or already capturing
[2024.12.08-03.04.32:096][405]LogTemp: Audio capture started
[2024.12.08-03.04.32:104][405]PIE: Server logged in
[2024.12.08-03.04.32:105][405]PIE: Play in editor total start time 0.718 seconds.
[2024.12.08-03.04.32:111][405]LogTemp: Audio out -> 479 samples
[2024.12.08-03.04.32:370][407]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.32:654][409]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.32:914][410]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.33:168][412]LogTemp: WebSocket connected
[2024.12.08-03.04.33:168][412]LogTemp: Sending Session Update Event: {
"type": "session.update",
"session": {
"modalities": ["text", "audio"],
"instructions": "My Name is Mayank, and he is very friendly and cheerful in nature.",
"voice": "shimmer",
"input_audio_format": "pcm16",
"output_audio_format": "pcm16",
"input_audio_transcription": {
"model": "whisper-1"
},
"turn_detection": {
"type": "server_vad",
"threshold": 0.500000,
"prefix_padding_ms": 300,
"silence_duration_ms": 500
},
"tools": [],
"tool_choice": "auto"
}
}
[2024.12.08-03.04.33:168][412]LogTemp: No create response message provided, skipping response create event
[2024.12.08-03.04.33:170][413]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.33:257][413]LogTemp: ===> Event Type: session.created
[2024.12.08-03.04.33:433][416]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.33:692][416]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.33:879][418]LogTemp: ===> Event Type: session.updated
[2024.12.08-03.04.33:952][421]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.34:210][423]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.34:472][426]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.34:732][429]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.34:993][432]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.37:330][516]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.37:592][528]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.37:853][540]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.38:113][551]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.38:370][563]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.38:633][575]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.38:894][587]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.39:152][598]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.51:372][167]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.51:633][179]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.53:970][284]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.54:232][295]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.54:493][307]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.54:753][319]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.55:010][330]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.55:273][342]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.55:533][354]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.55:793][365]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.56:050][377]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.56:312][388]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.56:572][400]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.56:832][411]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.57:090][423]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.57:352][434]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.57:612][446]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.57:872][458]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.58:130][469]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.58:393][481]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.58:653][493]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.58:912][504]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.04.59:170][516]LogTemp: Audio out -> 12454 samples
s
[2024.12.08-03.05.10:873][ 77]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.11:132][ 92]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.11:232][ 98]LogTemp: ===> Event Type: input_audio_buffer.speech_started
[2024.12.08-03.05.11:232][ 98]LogTemp: -------------------------------------------------> Response was cancelled due to turn_detected
[2024.12.08-03.05.11:248][ 99]LogRuntimeAudioImporter: Warning: The sound wave 'CapturableSoundWave_136' is not playing
[2024.12.08-03.05.11:250][ 99]LogEOSSDK: LogEOS: Updating Product SDK Config, Time: 18564.003906
[2024.12.08-03.05.11:393][108]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.11:499][115]LogTemp: ===> Event Type: input_audio_buffer.speech_stopped
[2024.12.08-03.05.11:532][116]LogTemp: ===> Event Type: input_audio_buffer.committed
[2024.12.08-03.05.11:532][116]LogEOSSDK: LogEOS: SDK Config Product Update Request Completed - No Change
[2024.12.08-03.05.11:532][116]LogEOSSDK: LogEOS: ScheduleNextSDKConfigDataUpdate - Time: 18564.273438, Update Interval: 353.215729
[2024.12.08-03.05.11:566][118]LogTemp: ===> Event Type: conversation.item.created
[2024.12.08-03.05.11:599][120]LogTemp: ===> Event Type: conversation.item.input_audio_transcription.completed
[2024.12.08-03.05.11:631][122]LogTemp: ===> Event Type: response.created
[2024.12.08-03.05.11:650][124]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.11:665][124]LogTemp: ===> Event Type: response.done
[2024.12.08-03.05.11:912][139]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.12:173][155]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.12:433][171]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.12:690][186]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.12:953][202]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.13:212][217]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.13:473][233]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.13:730][248]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.13:993][264]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.14:253][280]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.14:348][285]LogTemp: ===> Event Type: input_audio_buffer.speech_started
[2024.12.08-03.05.14:348][285]LogTemp: -------------------------------------------------> Response was cancelled due to turn_detected
[2024.12.08-03.05.14:366][286]LogRuntimeAudioImporter: Warning: The sound wave 'CapturableSoundWave_137' is not playing
[2024.12.08-03.05.14:512][295]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.14:614][301]LogTemp: ===> Event Type: input_audio_buffer.speech_stopped
[2024.12.08-03.05.14:649][303]LogTemp: ===> Event Type: input_audio_buffer.committed
[2024.12.08-03.05.14:681][305]LogTemp: ===> Event Type: conversation.item.created
[2024.12.08-03.05.14:714][307]LogTemp: ===> Event Type: conversation.item.input_audio_transcription.completed
[2024.12.08-03.05.14:747][309]LogTemp: ===> Event Type: response.created
[2024.12.08-03.05.14:770][311]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.14:781][311]LogTemp: ===> Event Type: response.done
[2024.12.08-03.05.15:032][327]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.15:293][342]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.15:553][358]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.15:810][373]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.16:072][389]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.16:333][405]LogTemp: Audio out -> 12454 samples
[2024.12.08-03.05.16:593][420]LogTemp: Audio out -> 12454 samples
This means the OpenAI Realtime API is detecting your speech so that part is working. Do you not year anything in the speakers on headphones? Do other sounds play in Unreal Editor?
1 Like
I have an echo issue when using the plugin and listening through speakers. Essentially, the voice feeds back into itself because the speakers’ sound is picked up by the microphone. I tried using a sound submix, but I can’t access the plugin’s audio capture component. What options are available in Blueprint to manage the audio capture component and assign the sound submix to handle the microphone? Thanks a lot
@robertb
I Do not hear any sound in the Laptop Microphone or Speaker.
I have Used Runtime Audio Importer Plugin, OpenAI Plugin to Create the Blueprint.
Do i need any Extra Tools to Encode or Decode the Audio?
Here is the audio capture code but yes this is a problem but we solved it i our POC demo with a directional mic. Here is the audio capture code: OpenAI-Api-Unreal/Source/OpenAIAPI/Private/OpenAIAudioCapture.cpp at main · rbjarnason/OpenAI-Api-Unreal · GitHub
1 Like
Do other Unreal Editor sounds play? If not then try to figure that out. This demo is my first project eve with Unreal Editor so I’m afraid I don’t know much about it in general.
1 Like
this is really cool, i wonder what you could do with a budget of a big studio and a team of ue5 devs. I’m sure the hardest part is to match the facial expressions and the phrase semantics, as well the body language that matches what is going on.
do you plan to further improve your ue5 project? if the real time endpoint wasn’t super expensive, it sure would be interesting to have a village or city interactive game/simulation like this. gotta love metahumans, but gpu’s aren’t the biggest fans. on the prompt engineering side of things, it would be nice to also add context on how it should sounds according to what is going on to avoid monotones, but hey, this is really cool, congrats on the project!
maybe 10 years from now, uh?
1 Like
hi robert, the open source version doesn’t seem to build on Unreal 5.3, maybe i’m not doing it right, but i’ve tried it a couple of times and it doesn’t seem to.
any idea on what i can do?
hi sam, what marketplace did you buy the plugin from?
1 Like