“Hello, I am currently working on a project and encountering an issue that I need assistance with. I am utilizing the ChatGPT Chat Completion API to stream data in chunks, which I then send to a React app from a Node Express server. This setup works fine in a local environment, where I am able to stream data effectively. However, after deploying to an AWS Lambda function, it no longer functions as expected; I’m receiving data in a single shot at the end instead of in chunks. To send data to the client, I use res.write("response from API in chunks") . How can I resolve this issue to ensure data is streamed in chunks on AWS as it is locally?”
Hey there and welcome!
Soooo, I’m not sure what this is:
All I know is that you need to set streaming to true in completions. I’m guessing you’re just using node.js then here? Are you using python to call the OpenAI client?
AWS services can get pretty complex, vast, and tricky real quick. Also, Lambda is serverless, so the traditional means of identifying the client/server relationships in your architecture to identify where the stream is bottlenecked does not apply here. If it were me, I’d prefer to just spin up direct servers with something like Terraform, but to each their own 
. Lambda may be well suited for this, I just don’t know because they have 10,000 different services that all try to do slightly different things.
You can show us you code so we can see if there’s any programming errors that might cause this. That will rule out those kinds of issues. I have a guess though that if it worked before and the instance is the problem, then it’s something to do with the way the Lambda/architecture is configured in AWS. If that’s the case though, not sure how we’d be of too much help here.
I say this because I never have any problems building things, but I always have problems configuring AWS servers. It’s like jumping into a decade’s worth of bells and whistles that seldom apply to anything you’re ever going to do.
1 Like