NodeJS multiple requests until it's complete?

Prompting
1

Hi,
My first GPT-3 project is to pull some basic JSON from a body of text. Using NodeJS.

In the Playground:

The completion works great. I have to keep pressing the “Submit button” though, to get it to finish.

Take at least 5 times. I guess this is because of the token limitations on each request.

In NodeJS:

Using the same prompt that worked in Playground (which required multiple Submits), would the equivalent way in a programatic NodeJS environment be to just keep submitting requests? Or like maybe partial requests? Maybe keep stringify-ing the concat’d completion until it is a valid JSON object?

Are there any pointers, articles, resources that could point me in the right direction for this sort of thing?

2

Welcome to the community!

What model and settings are you using. What do you have max_tokens set to?

1 Like
3

I’m using text-davinci-003

const { Configuration, OpenAIApi } = require("openai");

const configuration = new Configuration({
  apiKey: process.env.OPENAI_API_KEY,
});
const openai = new OpenAIApi(configuration);

const response = await openai.createCompletion({
  model: "text-davinci-003",
  prompt: "Summarize the TEXT below into a JSON with exactly the following structure { FirstName, LastName, Email }: <TEXT INSERTED HERE>",
  temperature: 0,
  max_tokens: 60,
  top_p: 0,
  frequency_penalty: 0,
  presence_penalty: 0,
});

I’m playing with max_tokens in hopes it will return the entire completion in one request.

EDIT: Yes it works!!

I pasting in the prompt above (including a 5000 character string) in the https://beta.openai.com/tokenizer tool.

It’s about 1,993 tokens.

So I bumped up the max_tokens to 2k.

But in cases where the input text a larger, pushing the total allowed tokens above 4k. I guess this where I’d need to do more than one request / completion? Not sure how to do that yet…

1 Like
4

Nice. Just remember that the 2k max tokens is for prompt+output. Max_tokens can somewhat control the output but not exactly all the time.

What you’ll likely need to do is chunk it up to 1k or 1.5k tokens and do it a bit at a time.

Good luck!

5

I think I see what you mean…

Split up the input text if it exceeds N characters? Then make separate requests?

E.G.
If user adds 5000 characters, split it up and make 5 requests with 1000 characters.

6

Yeah, or just limit user to 1000 tokens until the context window is larger…