# Batching prompts still being recommended despite not

**URL:** <https://community.openai.com/t/batching-prompts-still-being-recommended-despite-not/561552>\
**Category:** Documentation\
**Tags:** api\
**Created:** [December 16, 2023, 10:10pm UTC](https://community.openai.com/t/batching-prompts-still-being-recommended-despite-not/561552 "2023-12-16T22:10:11Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ashdragon](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/ashdragon/32/258814_2.png) [@ashdragon](https://community.openai.com/u/ashdragon)\
**Post date:** [December 16, 2023, 10:10pm UTC](https://community.openai.com/t/batching-prompts-still-being-recommended-despite-not/561552/1 "2023-12-16T22:10:11Z")

</div>

Batching prompts is not supported by the chat completions. There are a number of posts in the community forums complaining about this .  
But these two pages, which have been edited since Developer Day, still recommend batching prompts.

[https://platform.openai.com/docs/guides/rate-limits/error-mitigation](https://platform.openai.com/docs/guides/rate-limits/error-mitigation)

[https://platform.openai.com/docs/guides/production-best-practices/improving-latencies](https://platform.openai.com/docs/guides/production-best-practices/improving-latencies)

---

<div class="post-metadata">

**Author:** ![\_j](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/_j/32/766292_2.png) [@\_j](https://community.openai.com/u/_j)\
**Post date:** [December 16, 2023, 11:04pm UTC](https://community.openai.com/t/batching-prompts-still-being-recommended-despite-not/561552/2 "2023-12-16T23:04:41Z")

</div>

You have a good discovery of documents that have limited applicability - and show use of a model that will be turned off in weeks. Solution answering no particular problem.

```auto
prompts = ["Once upon a time,"] * num_stories
 
# batched example, with 10 story completions per request
response = client.completions.create(
    model="curie",
    prompt=prompts,
    max_tokens=20,
)

```

The information is useful if the endpoint and non-chat mode is made clear. It demonstrates _completion_ - having the AI continue writing where text is left open.

The example above is still poor, because it relies only on token sampling variations to not make ten near-identical stories that unfold exactly the same due to their independence. A confidently-trained low perplexity AI will have even more uniformity. This is better demonstrated with a list of inputs.

---

<div class="post-metadata">

**Author:** ![ashdragon](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/ashdragon/32/258814_2.png) [@ashdragon](https://community.openai.com/u/ashdragon)\
**Post date:** [December 16, 2023, 11:21pm UTC](https://community.openai.com/t/batching-prompts-still-being-recommended-despite-not/561552/3 "2023-12-16T23:21:11Z")

</div>

I was looking for ways to speed up API calls and found it. And found out the hard way that it didn’t work for chat completions

---

<div class="post-metadata">

**Author:** ![8020sllc](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/8020sllc/32/149945_2.png) [@8020sllc](https://community.openai.com/u/8020sllc)\
**Post date:** [January 23, 2024, 6:09pm UTC](https://community.openai.com/t/batching-prompts-still-being-recommended-despite-not/561552/4 "2024-01-23T18:09:35Z")

</div>

bump. recommend request batching capability be added to chat completions api. it seems they want to steer the api towards chat completions vs completions, and batches to ease high request volume, but have not put the two together yet.

---

<div class="post-metadata">

**Author:** ![Diet](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/diet/32/562749_2.png) [@Diet](https://community.openai.com/u/Diet)\
**Post date:** [January 23, 2024, 6:13pm UTC](https://community.openai.com/t/batching-prompts-still-being-recommended-despite-not/561552/5 "2024-01-23T18:13:54Z")

</div>

Are you hitting RPM limits, or would it be more of a convenience thing for you?

---

<div class="post-metadata">

**Author:** ![ashdragon](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/ashdragon/32/258814_2.png) [@ashdragon](https://community.openai.com/u/ashdragon)\
**Post date:** [January 23, 2024, 6:39pm UTC](https://community.openai.com/t/batching-prompts-still-being-recommended-despite-not/561552/6 "2024-01-23T18:39:46Z")

</div>

I implemented my own rate limiting into my code because I was occasionally hitting the RPM limit for GPT-3. 5

But my thought had been for the GPT-4 calls, if I could batch them, I could minimize latency of just waiting for data to move back and forth.  
The app I’m working on has 50k-200k tokens of data per run split into 3000-5000 token chunks, and takes about 20 minutes just for the GPT-4 calls. If I could batch, I could probably shave off a minute or two. Not a huge amount, but still something.