I’d recommend checking the service_tier field in the Responses objects you received with higher latencies. This would indicate which service tier that particular request was processed under.
You are getting rate limited. I have a test suite I built that will do a few hundred API calls. Some stall out like this, then continue like normal. Depending on your use case, you can use a timer and resend the request. I helped, but did not fully fix, my issues by just funding my account to get to the next tier. Also, I noticed that the wording is a little ambiguous on the usage tier. You do not need to “use” the funds, just fund your account to get to the next tier.