# Follow-up Inquiry on Realtime API Issues in AI Interviewer Implementation

**URL:** <https://community.openai.com/t/follow-up-inquiry-on-realtime-api-issues-in-ai-interviewer-implementation/1040529>\
**Category:** Bugs\
**Created:** [December 3, 2024, 1:34am UTC](https://community.openai.com/t/follow-up-inquiry-on-realtime-api-issues-in-ai-interviewer-implementation/1040529 "2024-12-03T01:34:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![hello-b](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/hello-b/32/448459_2.png) [@hello-b](https://community.openai.com/u/hello-b)\
**Post date:** [December 3, 2024, 1:34am UTC](https://community.openai.com/t/follow-up-inquiry-on-realtime-api-issues-in-ai-interviewer-implementation/1040529/1 "2024-12-03T01:34:39Z")

</div>

I am writing to follow up on specific challenges we have encountered while utilizing the OpenAI Realtime API to develop an AI Interviewer. The system is designed to simulate professional job interviews, incorporating a dynamic interview guide and Voice Activity Detection (VAD) for real-time interaction. To assist in diagnosing and resolving the issues, we are providing the following details about our configuration and behavior expectations.

**Configuration Overview**

We use the following session configuration:

```auto
realtimeClient.updateSession({
    turn_detection: {
        threshold: 0.5,
        prefix_padding_ms: 300,
        silence_duration_ms: 600,
        type: 'server_vad'
    },
    modalities: ['text', 'audio'],
    input_audio_format: 'pcm16',
    output_audio_format: 'pcm16',
    input_audio_transcription: { model: 'whisper-1' },
    instructions,
    voice
});

```

**Issues Encountered**

1. **Language Mismatch**

Despite specifying a particular interview language (e.g., English), responses sometimes shift to other languages, likely influenced by dynamic resume data. We strictly intend responses to follow the specified language setting.

- **Question:** Is there a recommended way to enforce consistent language usage, even with multilingual input data?

1. **Noise Sensitivity and VAD Behavior**

The VAD implementation occasionally interprets background noise as valid input, interrupting active audio streams or triggering unnecessary responses.

- **Question:** Could you suggest best practices or configuration adjustments to reduce noise sensitivity and enhance VAD accuracy?

1. **Self-Generated Responses**

The AI sometimes initiates self-dialogue—posing questions and immediately answering them without processing actual user input. This deviates from the expected interaction flow.

- **Question:** Are there potential misconfigurations or known API behaviors that could lead to this?

1. **Truncated Audio Playback**

Audio responses occasionally cut off toward the end, even though the corresponding transcribed text appears complete.

- **Question:** Could this result from API limitations, streaming inconsistencies, or an incorrect configuration?

1. **Speech Recognition Accuracy**

There is a noticeable discrepancy between the transcribed text and the actual spoken input. Non-verbal sounds like coughing or sighing are sometimes interpreted as valid phrases (e.g., “Hello” or “OK”), triggering unintended responses.

- **Question:** Are there configuration adjustments or additional filters to improve transcription accuracy and reduce misinterpretation of non-verbal sounds?

**Request for Assistance**

We kindly request the following:

- Clarifications or solutions for the issues described above.

- Any relevant documentation to improve our understanding and usage of the Realtime API in this context.

- Insights into whether these issues could stem from the Realtime API’s Beta phase and the likelihood of encountering such challenges.

We appreciate any guidance you can provide to enhance the reliability of our AI Interviewer and ensure a seamless user experience. If necessary, we are happy to share additional configuration details or logs for further investigation.

To aid in your analysis, we have attached a version of the prompt instructions with sensitive information redacted.

Thank you for your support, and we look forward to your response.

---

<div class="post-metadata">

**Author:** ![Shinyu](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/shinyu/32/206332_2.png) [@Shinyu](https://community.openai.com/u/Shinyu)\
**Post date:** [December 6, 2024, 1:50am UTC](https://community.openai.com/t/follow-up-inquiry-on-realtime-api-issues-in-ai-interviewer-implementation/1040529/2 "2024-12-06T01:50:30Z")

</div>

I have same issue with 2 and 5.  
For 2, the noise sensitivity part is extremely annoying. For example, someone is calling with music playing in background. Sometimes the model will process the music input when the user is silent. I hope the model can focus on the initial recognized voice for the entire session.  
For 5, I find the model weirdly recognize name badly. I have a few use cases from different users, they clearly say Jordan. And the model say their name is Mark, etc.