Hi,
I have a task for which I need to call the completions API several thousand times and expect this to take several hours.
I don’t need help with optimising the prompt itself, but to understand what’s the most recommended way to parallelise (if this is allowed) the calls to the API so that the task completes a little faster.
Also, could anyone confirm if there is a limit to the number of concurrent calls? Some information I’ve found on the internet suggests that only up to two concurrent requests are possible.
Any other solution that could help complete the task faster with respect to making the API requests would be appreciated. Thanks!
Hello, it turns out that the OpenAI’s completion call can automatically handle the multiple request and all runs in parallel.
So no need to use python’s native async libraries!
prompt_1 = "This is the prompt - 1 to run separately"
prompt_2 = "This is the prompt - 2 to run separately"
prompt_3 = "This is the prompt - 3 to run separately"
prompts = [prompt_1, prompt_2, prompt_3]
response = openai.Completion.create(
model="text-davinci-003",
prompt=prompts,
max_tokens=128,
temperature=0.7,
)
# Print the responses
for choice in response.choices:
print(choice.text, end="\n---\n")
Warning: the response object may not return completions in the order of the prompts, so always remember to match responses back to prompts using the index field.
That’s it! It all runs in parallel! Documentation OpenAI Platform
Sending in a batch of prompts works exactly the same as a normal API call, except you pass in a list of strings to the prompt parameter instead of a single string.
Completion models take strings.
ChatCompletion model endpoints instead take a list of dictionaries.
A list of a list of dictionaries is not permitted. Trial:
messages=[
[
{
"role": "system",
"content": """You are an assistant.""",
},
{
"role": "user",
"content": "What is the capitol of California?",
},
],
[
{
"role": "system",
"content": """You are an assistant.""",
},
{
"role": "user",
"content": "What is the capitol of New York?",
},
]
]
= fail:
“API Error: [{‘role’: ‘system’, ‘content’: ‘You are an assistant.’}, {‘role’: ‘user’, ‘content’: ‘What is the capitol of California?’}] is not of type ‘object’ - ‘messages.0’”
Warning: the response object may not return completions in the order of the prompts, so always remember to match responses back to prompts using the index field.
Documentation 