So I developed a web app that access the OpenAI api with a prompt inputted by the user. It took around a minute and a half for response to comeback. Now this worked fine on my local machine but when I went to host the web app on heroku I get request timed out. Any ideas on what I could do to fix this? I’ve looked into running the api call asynchronously with celery, but I’m wondering if there is a better way?
1 Like
Look into websockets and workers. Heroku provides support for both of these things, and they’ll allow you to get around the 30 second timeout on heroku.
Would streaming the response back also work around this timeout?
It should do, yes. In my test setup I get the first response token after a few hundred milliseconds.
If the latency is acceptable (and you don’t want to stream) then you could also host the API call in an AWS Lambda function, and input is a webhook over API Gateway.
Timeout is 15 minutes if you increase it from the default 3 seconds 
could you point me to some documentation on setting this up? Thank you!
1 Like
Awesome! If I can use workers in the background that would be great for not getting timed out each time at heroku when waiting for the response from openai which takes more than 30s.
Thanks, will try to put in the workers.
Can - running into the same issue. Have you solved it?