I would like to make a request to the OpenAI API for completion when typing in an input field with a 500ms debounce delay. Typically, if a user types a letter into the input field, a request will be made. If they quickly type another letter, there is an ongoing request that we need to cancel. How can I do this? Can I abort the API request using the JavaScript AbortController? However, I am still concerned that even if I abort the request, the completion process continues in the backend and costs are deducted from my balance. If this is the case, what other methods can I use to meet my requirements?
Completions will be aborted, and consequentially no further tokens will be billed, if and only if you use streaming responses and close the connection.
Note that even if you abort your stream, you will still be charged for the context every time.
You can consider eating the cost.
Note that this will be expensive in any situation: your user will squat/block your VRAM, without actually producing any useful output.
You could use a separate model to try to predict whether it’s worthwhile trying to answer the user query before actually going into generation. But if you used embeddings for this, you might as well use a generation model, and restart the generation if the predicted user output deviates too far from what the user is actually entering
But a submit button would probably be the easiest option!
