That button only disables “legacy user keys”, the older organization owner-based key.
What is available does not stop the usage, and there is no way to terminate with a spending limit except by monitoring, and intentional degrading of the service made months ago.
Setting a monthly budget allows you to establish soft spending thresholds for your project. When usage exceeds this limit within a given calendar month (UTC), API requests will continue to be processed without interruption. This feature is designed to help you monitor usage through budget alerts, but it does not enforce a hard cap on spending.
The best you can do, based on monthly usage (meaning the start of the month would be a key hacker’s best attack surface)
OpenAI will shut you off by your tier’s monthly hard limit, so they can do it with technology, yet they removed it from the API for you, so your account can be emptied.