In Tefeno we developed some applications that utilise GPT-3, we are facing common struggles:
- Cannot create pre-signed requests: we use serverless architecture which does not support streaming. Therefore, if the response of GPT-3 is long… the waiting time for the client is also long as the entire request has to be loaded before sending the response. We cannot show the prompt details as well as send back the secrets to the client. We do not want our services to act as gateways to openai.
Solution: generate a pre-signed request by providing the prompt options and the openai key, our services will return this URL and the client or the browser can directly send the request to openai. OpenAI will quickly decrypt this request using its private key and perform the request normally. This way the client can directly read the stream from openai and read the generated text one at a time reducing loading time and enhancing user experience.
- Usage not tracked per access key and cannot figure out which is which: since we have many apps using different keys… we have to measure ourselves in a database the usage whereas this could have been done automatically by openai. Furthermore, if we want to delete keys such as development keys… we cannot figure out which is which and have to check the codebase to correctly delete the right keys.
Hope this helps!
