prompts = [
“Prompt 1 goes here.”,
“Prompt 2 goes here.”,
“Prompt 3 goes here.”
]
response = openai.ChatCompletion.create(
model=“gpt-4”,
messages=[
{“role”: “user”, “content”: prompt},
],
max_tokens=750,
)
prompts are 100 in number. How I can manage this?
Hi there!
Thanks for your question. Can you maybe take a step back and try to explain what you are trying to achieve and/or provide an example?
As part of a prompt it is generally possible to provide longer instructions, i.e. multiple sentences. So perhaps what you are trying to achieve is possible. But in the absence of more specific information, it’s a bit difficult at the moment to provide you with further guidance.
Yes sure.
So I am generating emails with open ai chat api with around 100 different prompts.
If I generate it one by one It will take around 10 mins.
So I want to you is it’s possible to do that in asynchronous way.
You are limited by the number of requests per second. But generally just use asyncio in python to accomplidh the same?
Hi!
I’ll make a bold assumption that you will hit your rate limits first.
But you can avoid that and achieve your goal with batching the requests:
https://platform.openai.com/docs/guides/rate-limits/batching-requests
Oh thanks a lot I will try to implement this approach.
Thanks 
Hi @vb
I’ve successfully implemented batching for completions.create, but I’m encountering difficulties when trying to implement batching for chat completions. Could you provide guidance on how to extend batching to chat completions as well?
Good to hear that you are making progress but I don’t understand what actual issue is.
completion = client.chat.completions.create(
As chat.completions is the endpoint and create is the method.
Hi @vb
I have implemented batching with the completions method using the https://api.openai.com/v1/completions endpoint. However, I encountered issues with the response not meeting my requirements. Therefore, I switched to using the https://api.openai.com/v1/chat/completions endpoint. Unfortunately, I haven’t found a way to implement batching with this endpoint.
can you please help me with that.
Yeah, let’s start with you sharing what you have tried and what the results are, what the issue is etc…
For example you can share your code.
Asyncio with Parallel calls = all your data at once returned in smallest time frame.
Here this will help many:
async def fetch_data_and_summarize(self, username):
try:
# Parallel fetch operations
pets_data, preferences_data, people_data, relationship_data, mental_health_data, physical_health_data = await asyncio.gather(
self.agent.fetch_pets_data_for_user(username),
self.agent.fetch_preferences_data_for_user(username),
self.agent.fetch_people_data_for_user(username),
self.agent.fetch_relationship_data_for_user(username),
self.agent.fetch_mental_health_data_for_user(username),
self.agent.fetch_physical_health_data_for_user(username)
)
# Generate summaries for each category using GPT-3.5-turbo
summaries = await asyncio.gather(
self._generate_summary(pets_data, "pets", username),
self._generate_summary(preferences_data, "preferences"),
self._generate_summary(people_data, "people"),
self._generate_summary(relationship_data, "relationship"),
self._generate_health_summary(mental_health_data, physical_health_data)
)
summaries_dict = {"pets": summaries[0], "preferences": summaries[1], "people": summaries[2], "relationship": summaries[3], "health": summaries[4]}
await self.agent.update_user_profile_with_summaries(username, summaries_dict)
except Exception as e:
logging.error(f"Error summarizing data for user {username}: {e}")
async def _generate_summary(self, data, category, username=None):
if not data:
return "No data available."
if category == "people" or category == "pets":
# Adjusting prompt for detailed list generation
prompt = f"Given the following data for {category}, generate a detailed list including every known detail about each entity. Ensure no entity is omitted and provide insights that encapsulate their characteristics, relationships, and any notable anecdotes. {category.capitalize()} data for user {username}: {data}"
else:
prompt = f"Please summarize the following data to extract key insights and provide a comprehensive understanding as a data point for the AI system, minimizing duplication and maximizing clarity: [{category}]: {data}"
try:
response = await self.async_client.chat.completions.create(
model="gpt-3.5-turbo",
messages=[{"role": "system", "content": prompt}],
max_tokens=200 # Increased token count for more detailed summaries
)
if response.choices:
return response.choices[0].message.content.strip()
else:
logging.error(f"No valid choice in the OpenAI response for {category}")
return f"Summary generation failed for {category}."
except Exception as e:
logging.error(f"Error generating summary for {category}: {e}")
return f"Summary generation failed for {category}."
Basically this is a simple category system that calls the same function to summarize based on the classification.
you can also do same concept using a list of prompts and passing each through the same system in similar way to get what you want. be sure to check out kruel.ai a full working memex system on gpt3.5 16k token model as it’s the cheapest model to get the most bang for the $
Works for me.
@rajatkhanna801 you can mark this topic as solved, if this is what you have been looking for.
Probably not.
The issue, ofc, is that completion api incorporates batch within itself. Meaning that you can get multiple choices, not just choice[0].
The advantage of multiple choices is that even if you are limited by API Calls /minute, you can batch multiple calls within completion; presuming that you have enough tokens.
Chatcompletion on the otherhand, can only do one at a time. So you hit the API Calls per minute limit much sooner.
In short completion increases your bandwidth.
It’s really just about sending several different requests at the same time instead of doing this consecutively.
From the original question:
prompts are 100 in number.
and this is covered even by Tier 1.
You can hit the rate limit prematurely by shooting of a ton of requests at the same time. Requests per minute can also be considered to be similar to requests per second * 60.
This gives a framework for making rate-limited parallel requests to the API, manually specifying the endpoint instead of using the openai library.
It could be adapted to be more straightforward and written around openai library, focused on chat completions with a system message that performs the same scope of operations on multiple inputs, and with a straightforward input file and a resumable disk database of responses. It also could be improved to take the x-ratelimit headers and tune the limit before getting API denials.
My Issue is still not resolved
Here is my code
import asyncio
import aiohttp
import openai
import time
openai.api_key = api_key
async def api_request(prompt):
async with aiohttp.ClientSession() as session:
response = await session.post(
‘/v1/chat/completions’,
json={
“model”: “gpt-4”,
“messages”:[{“role”: “user”, “content”: prompt}],
“temperature”: 0.75,
“max_tokens”: 750
},
headers={‘Authorization’: f’Bearer {openai.api_key}'}
)
return await response.json()
async def main():
tasks = [api_request(prompt) for prompt in prompts[:20]]
completions = await asyncio.gather(*tasks)
for completion in completions:
print(completion[‘choices’][0][‘message’][‘content’])
if name == “main”:
start_time = time.time()
asyncio.run(main())
completion_time = time.time() - start_time
print(“completion_time===============”, completion_time)
I just trying with 20 different prompts and still APi is showing conmection timeout error.
Please let me know if you anything with that.