If I am calling OpenAI API from an AWS Lambda node.js code (or actually any server code), the server code needs to wait until it gets the reply from the OpenAI API. The response can take tens of seconds. During this time the server code must be running so I am paying for the execution time even if I would await for the execution of the API call in an asychronous way. The Lambda code must be running and holding the connection to the OpenAI API.
Currently I am using PHP for calling the API. There the script must run all the time and wait for the API call to complete. So I spend quite a lot of server resources since the waiting is long. Is there a way how to workaround this? If OpenAI had an asychronous API where you would send a request and could poll for the response, that would save a lot of server time. Is there a solution like this?
1 Like
Perhaps lambdas/functions might not be the best choice for you.
I don’t remember the exact numbers but IIRC node can handle tens (if not hundreds) of thousands of parallel connections in a single instance. spawning a new container instance for every request is obviously a huge waste.
If it’s technically too involved for you /don’t wanna deal with it, maybe consider using one of those bot building platforms.
edit: cloud foundry used to be really good for stuff like this, but it seems like cf seems to be slowly becoming less and less accessible.
The API is 100% synchronous right now.
To get something back instantly, look into streaming through the API.