Whisper API - subtitle timecodes out of sync

I am using Whisper API to transcribe texts and it works well, even with smaller languages. However, I am having problems with transcribing subtitles, as it will happen relatively frequently, that subtitles will go out of sync in some videos. I have noticed that English transcriptions might fare better.

Did anyone have experience with this and can confirm this problem exists? Is there a way to solve it?