Cant setup streaming for code-gen

API
1

Below is my code getter after its being generated, and the request handler.

export const getTransformedCode = async (query, res) => {
  const prompt = instruction(query);
  const response = await openai.chat.completions.create({
    messages: [{ role: "user", content: prompt }],
    model: "gpt-4",
    stream: true,
  });

  try {
    for await (const chunk of response) {
      console.log("Streaming data:", chunk.choices[0].delta.content);
      if (chunk.choices[0].delta && chunk.choices[0].delta.content) {
        res.write(chunk.choices[0].delta.content);
      }
    }
    res.end();
  } catch (error) {
    console.error("Error during streaming:", error);
    if (!res.headersSent) {
      res.status(500).json({ error: "Error during streaming data" });
    } else {
      res.end();
    }
  }
};

export default async function handler(req, res) {
  if (req.method === "POST") {
    try {
      const { prompt } = req.body;
      res.writeHead(200, { "Content-Type": "text/plain" });
      await getTransformedCode(prompt, res);
    } catch (error) {
      console.error("Error in handler:", error);
      res.status(500).json({ error: "Internal server error" });
    }
  } else {
    res.status(404).json({ error: "Not found" });
  }
}

If I call a mock request via Postman I can see many chunks for the console log “Streaming data: chunk.choices[0].delta.content”, which means the for loop on the backend is looping properly and chunks are accessed properly.

I am sending the request from the frontend, from here:

export const callOpenAI = async (
  prompt: string,
  signal: AbortSignal,
  onChunkReceived: (chunk: string) => void // Callback to handle each chunk
): Promise<void> => {
  try {
    const response = await fetch("/api/completion", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ prompt }),
      signal: signal,
    });

    if (!response.ok) {
      const errorText = await response.text();
      throw new Error(`API Error: ${errorText}`);
    }

    if (!response.body) {
      throw new Error("Response body is null");
    }
    const reader = response.body.getReader();
    const decoder = new TextDecoder("utf-8");

    let done, value;
    while (!done) {
      ({ done, value } = await reader.read());
      if (value) {
        const chunk = decoder.decode(value);
        console.log(chunk);
        onChunkReceived(chunk);
      }
    }
    reader.releaseLock();
  } catch (error: any) {
    if (error.name === "AbortError") {
      console.log("Code generation aborted ", error.message);
    } else {
      console.error("Error communicating with OpenAI:", error.message);
    }
    throw error;
  }
};

the while loop is not looping its console.logging just 1 chunk which is the whole code after it’s being generated by my AI assistant. What I want is for the while loop to work properly by console logging each chunk.

Thanks for the help.

2

Since you are using NodeJS & some sort of Javascript you can benefit using the Streams API