Integrating whisper API in front end

I asked my dev team to integrate whisper API for speech to text in our AI Agent app ( only on web). I was advised that front end integration creates security risks by exposing the API key and backend integration ( which is safer ) is complicated and need to be engineered properly to deal with time lag / latency it may create! This really compromises our Agent app - any suggestions?
FYI we are using assistant API and vector database at backend.