Feature Request: Major Improvements to Voice Mode Audio Handling / Scene Awareness
As a very heavy daily ChatGPT voice user, I think one of the biggest opportunities for improving user experience is not necessarily increasing model intelligence, but improving the audio interaction layer.
Current voice mode is already extremely impressive, but audio handling still feels like one of the main immersion-breaking bottlenecks.
Core issue
Voice mode is still overly disrupted by background sounds such as:
- passing vehicles
- ticking clocks / watches
- glasses clinking
- birds
- doors closing
- fans / appliances
- brief environmental noises
Even relatively minor transient sounds can cause:
- awkward pauses
- response resets
- broken conversational flow
- “Sorry, I heard background noise” interruptions
- partial loss of user context
For long-form conversations, this becomes frustrating.
Suggested architectural improvement
Instead of treating all incoming audio as a single stream to filter aggressively, consider splitting processing into two parallel pipelines:
1. Foreground Speech Pipeline
Dedicated to:
- isolating the user’s voice
- maintaining continuous semantic understanding
- preserving conversation flow even during transient noise
This pipeline should prioritize continuity.
The assistant should keep listening to the user unless speech itself becomes unintelligible.
2. Background Scene Analysis Pipeline
Process non-speech audio separately for environmental understanding.
This would classify sounds such as:
- traffic
- birds
- weather
- engines
- household appliances
- music
- animals
The assistant could optionally reference these naturally.
Example:
“Was that a blackbird?”
or
“Sounds like you’re outdoors.”
This would dramatically increase immersion and presence.
Why this matters
Humans don’t merely filter background noise.
We perform scene understanding.
We can simultaneously:
- follow speech
- ignore irrelevant noise
- identify meaningful sounds
- comment naturally when appropriate
That ability is central to natural conversation.
Current AI voice systems still feel fragile here.
It would be great though to sub-filter and capture… I mean theyve done incredible things on realtime video which is x10000 the bandwidth than audio ..