All the questions addressed by the API team during the December 17, 2024 AMA

[VeitB]

Link to the original topic with all questions and answers:

AMA on the 17th of December with OpenAI's API Team: Post Your Questions Here

Little Dev Day confirmed. Join the AMA with the API team right after the release! To prepare for the event—or if you’re unable to attend—you can post your questions here in advance. The AMA will begin at approximately 2024-12-17T18:30:00Z (UTC), right after the Shipmas live stream. You can also follow @OpenAIDevs on X (Twitter) to learn when the event will start. https://x.com/OpenAIDevs We’re looking forward to your questions, helpful answers, and an all-around great time! See you there!

Question:

@louzell

Would love to hear from the API team if the endless newline bug was fixed with streaming structured outputs, where the API spits out newlines for 90 seconds and then disconnects. Gif of it here. In the community forums here. Thanks!

@michellep

Hey Lou – sorry about this unfortunate behavior. This can happen when the model outputs are sufficiently improbable so the only valid token per your schema is the newline token. It’s not really a bug in that engineering changes cannot fix it. When this happens, I would suggest trying to use a simpler schema or changing your prompting so it’s more clear to the model what it should respond with. Let me know if this helps!

Question:

@xbaha

Will SORA have an API, if yes When?

@romainhuet

We don’t have plans for a Sora API yet, but we’d love to hear more! What will you build if we ship one?

Question:

@misreadable_mind

Assistants API for over a year now in BETA. Are the Assistants API dead-end? is it just proof of technology?

@romainhuet

We’re seeing a lot of developers building with the Assistants API, and we’re continuing to invest in the tools developers need to build agents. Next year, we plan to bring o1 to the Assistants API and will have more to share soon!

Question:

@mask.usa

At the moment, I am interested in information on the following questions:

1. Updating the API whisper on V3?
   The current version of V2 is not as accurate and contains hallucinations.
2. Expanding the capabilities for speech control in the API TTS? (pauses, emotions, etc.)
3. Voice cloning

@jeffsharris

Nothing to share yet on V3 Whisper in the API. But for both audio understanding and TTS, do check out the new GPT-4o mini audio preview model. It’s got state of the art speech understanding and you can prompt the model directly to control how it hears and speaks! For example, give it a prompt like "Say the following in a somber tone, and make sure to pause your speech appropriately: "

Question:

@sbd.ebine

When do you increase the maximum TOKEN (input) in the API? We are developing a data analysis tool and this is a must. As it is now, it will be replaced by gemini.

@jeffsharris

It’s something we care about! Giving the model more context and examples is a great way to get smarter responses. Nothing to announce just yet but stay tuned in 2025

Question:

@arieltro78

is there an option to send image frames to realtime api like in the new vision mode in advanced mode ? i am working on smart glasses for blind

@jeffsharris

Not yet! The model has been trained for this capability (you can play with it in ChatGPT now!). Stay tuned for it coming to API next year

Question:

@gp

I’ve tried the Realtime API a few days ago in Portuguese and unfortunately the results were pretty bad. I assume it uses whisper small under the hood which makes it useless in Portuguese. Should we expect a better performance with todays release?

@jeffsharris

It’s definitely worth retrying! Both GPT-4o and 4o mini have improved meaningfully in multi-lingual understanding with the latest snapshots. We still use the same Whisper model to transcribe what the user said, but then GPT-4o processes the audio directly and responds (without going through a transcription). Would love to hear what you find?

Question:

@astelmach20

Can there be an easier way of doing continuation (with structured outputs and streaming)? Right now it’s rather difficult to stitch 2 assistant responses appropriately

@michellep

Interesting question, and not one I’ve encountered before! It really depends on your schema and what you’re looking for. You may want to consider a different schema for your second response, which has something like a “continuation” key to make it clear to the model that it is supposed to add to its earlier response.

Question:

@trenton.dambrowitz

Great news, devs have been waiting patiently for our Shipmas present!

With o1 now having vision capabilities (which I presume will be released in the API in the future) how should we be thinking about using it for use-cases that typically require in-context learning?
Is it effective to provide the model a multi-shot prompt with each example having the ∼5 images that will be encountered in the real world, or is a different strategy recommended?

@michellep

We launched o1 with vision inputs today! Yes, we’ve seen few shot prompting to be quite effective to explain to the model how it should respond to new images.

Question:

@dowpenerd

Im sorry if this question has been asked already, and I absolutely appreciate you all and the accomplishment you have attributed to so many communities and the world as a whole. However I’m wondering if o1 is now available in Playground under Chat? Because at this time it’s only showing o1 preview still.

@edwinarbus

Which tier are you on? o1 has started rolling out to developers on usage tier 5 today, and rollout will complete after a few weeks. Team’s working on adding expanding to more tiers. (You’ll get an email notifying you once it’s available to you in the API and Playground.)

Question:

@sam.saffron

• What is the API team’s take on Model Context Protocol · GitHub by Anthropic?
• Dall-E seed parameter, pretty please! We need to be able to generate consistent images.
• Output vision improvements in GPT-4o and future models. Specifically, the most important capability would be finding (x,y) coordinates in a specific image.

@michellep

• It’s a cool protocol!
• For DALL-E, thanks for the input!
• I hear you on the vision improvements. We’re working on something in this space now and hope to have some improved vision models soon.

Question:

@james01

Is an ephemeral API key subject to abuse? For example, could a bad actor scrape a website every minute to keep getting new ephemeral keys to use for their own purposes?

@juberti

The keys are locked to a specific Realtime API session, so their use is substantially limited compared to typical OpenAI API keys.

Question:

@vicentes

can preference tuning be used to encourage longer answers (say 10+ pages) and better use of output context ?

@andrewpeng

Preference fine-tuning can encourage longer responses, however we still have a maximum number of output content tokens for each model.

Question:

@william.allan

I had a look at the docs for the new ‘developer’ message in the chat completions. I’m not quite clear on if you handle the system vs. developer message at your end, or if it’s something we as developers still need to handle via if 'o1' in model_name: do this else: do that kinda thing? And a follow-up if I may - there was a mention of work you’d been doing on the relative ‘weighting’ of system vs developer vs user messages - is that documented? I’d be really interested in digging in if it was! Thanks!

P.S No PHP SDK. The unloved and yet massively in-production step-child of the developer world. Santa? Can you hear me?..

@romainhuet

Great question on the developer message! The best resource to read more is our Model Spec detailing the hierarchy:

Follow the chain of command. Subject to its rules, the Model Spec explicitly delegates all remaining power to the developer (for API use cases) and end user. In some cases, the user and developer will provide conflicting instructions; in such cases, the developer message should take precedence.

Thank you for the note on PHP! With the addition of Java and Go today, our goal is to ultimately support the most popular languages and tech stacks. No timeline to share yet for a PHP SDK, but in the meantime, we recommend community-supported libraries to get started.

Question:

@anon25271712

What was the most difficult task you’ve worked on so far at OpenAI? What is the most fun task you’ve worked on so far?

@michellep

Building Structured Outputs was one of my personal highlights! The engineering side was actually super challenging, in order to get constraining to work at scale but also without impacting latency. There was a ton of work there to bring everything together in a performant way. If you want to learn more about it, we talked about it here: https://www.youtube.com/watch?v=kE4BkATIl9c

Question:

@356

Is it possible to do both supervised and preference fine tuning on a model?

Can you do preference fine tuning on an existing model fine tuned with supervision?

@andrewpeng

Yes! It is possible to do supervised fine-tuning and then run preference fine-tuning. We have seen good results from running SFT first and then DPO after.

Question:

@morietschel

Hi there!
As I am preparing for a rollout of o1 fine-tuning, how can I best prepare data, and build an intution, for this new mechanism?
My applciation is not gene matching; is there a way to conceptually guide / reward thinking steps?

@sherwinwu

We are actively working on a guide around how to approach reinforcement fine-tuning, and will publish it once it’s ready! But there are still some details we wanted to work out first.

At a high level though, I would say to keep several things in mind:

1. The underlying dataset are “tasks” – a set of instructions paired with an output that is the result of the task
2. Make sure the task is autogradable via the options we make available in the API (it should be easy to verify if the task was done correctly or not). We currently support a set of graders, but will expand over time. I know this is a bit hard because you can’t see the set of graders available until we launch more broadly – but easily gradable tasks (i.e. string match) are more likely to work out of the box.
3. Make sure that the task is clear enough that if expert humans do it, they also converge onto the same answer.

More to come here soon!

Question:

@surfsunset83

Hi there, what micro controller was used for the fawn?

Thanks,

Kyle

@juberti

The microcontroller was an ESP32-S3.

@Sean-Der

For the demo I used a https://sonatino.com. I have used the Realtime API Embedded SDK on multiple ESP32-S3 devices. You might even be able to use it in other ESP32s, performance wise I had a decent amount of overhead left.

Question:

@janalida

When will you be including search capabilities in the API.

@sherwinwu

We’ve heard a lot of demand for search and browse capabilities in the API – especially from folks who want to build applications that are grounded in super recent information. We’re actively working on this and expect to launch something next year!

Question:

@P0mme

Great work, team!

Do you plan to keep lowering the cost of the realtime API? A 60% price drop is simply amazing but it keeps the costs too high for some of our products.

@sherwinwu

We are always working on decreasing the prices of our models – so I would expect more here over time as well! I would also recommend trying the new 4o-mini realtime model we launched today, it’s quite a bit cheaper too.

Question:

@rxeconomics

On structured outputs,

1. Should we be confident building products with client.beta.chat.completions.parse for Pydantic structured outputs? Or go the JSON schema route?
2. Will o1-mini API soon be able to do structured outputs?

@brianz-oai

Yes, you should be confident building with client.beta.chat.completions.parse for Pydantic Structured Outputs!

Also yes, we are working on bringing structured outputs (and also other useful API features) to all reasoning models, including the o-mini series. Stay tuned.

Question:

@SomeUser2022

In the Playground, the beta feature for designing a json schema is really useful. Could it be given to us as an endpoint? Maybe we could then hook it up to a custom GPT as a function call so we can “chat with” our responseFormat.

@sherwinwu

If you’re referring to the “generate” feature that auto-generates a json schema for you – this is just a simple call to our chat completions API with a specific prompt! You can actually see the prompt we use in our documentation: https://platform.openai.com/docs/guides/prompt-generation#meta-schemas

Question:

@brendan.whiting

• Do we expect o1 and the realtime API to eventually converge? Or if we want an audio model that’s smart, will we need to call o1 as a tool call from the realtime API?
• Related: Do you have it on the roadmap for the realtime API to take video input? Or the o1 API to take audio or video input (similar to Gemini 2.0)?

@jeffsharris

Today, the best way is with a tool call that you use to trigger o1 (probably using the new out of band conversation feature). We’ll keep investing in making it easier to use more intelligence within the Realtime API

Re video. Stay tuned it’ll come next year. The model has been trained for this capability so in the mean time, you can experiment with it in ChatGPT

Question:

@rxeconomics

Any plans to make prompt caching better? Specifically, allow specification of what content to cache, give it a variable name, and allow reference to the cache in subsequent API calls for a certain time duration. I know another company is doing this

@jeffsharris

Nothing to share just yet. We focused the first pass of prompt caching on making it as easy as possible to use (no API changes needed, no extra cost for cache writes). It’s a feature we care a lot about though … would be curious how you’d want to use a more structured cache?

Question:

@henrik.hoeltzermann

Where can I buy the Reindeer that speaks?

@Sean-Der

The demo itself used a https://sonatino.com. You could use any ESP32-S3 though!

It was connected to Sennheiser Enterprise Solution SP 20 ML that I got off eBay. You can just use any headphones though.

The stuffed toy is Record Your Own Plush 16 inch Reindeer - Ready 2 Love in a Few Easy Steps - Walmart.com

Question:

@j0rdan

Great updates on Realtime API.

• Any timeline as to when the issue with the realtime model dropping out of voice mode when reconstructing conversation history is going to get resolved?
• Also the audio getting randomly cutoff at the end from time to time.
  These two are probably the most critical issues of the API right now.

@jeffsharris

We made a bunch of improvements to audio truncations with the new models… would be curious if you still see problems there?

It’s still possible to get the model stuck in text only mode if you give it a huge amount of text upfront. Known issue that we’ll keep improving in future releases. In the mean time, putting audio in the latest user turns can help the model rediscover its voice

Question:

@derk22

• will OpenAI also support the “prefill” for greater output control that Claude already has? It seems quite easy to implement, or is there a reason OpenAI doesnot support this?
• for the direct preference optimization (DPO), is it better to sample the preferred response and the non preferred response from the model that is finetuned or does it not matter where they come from. It seems more logical to sample them from the same model, but it can be hard to get 2 responses in a clear different style.

@andrewpeng

We don’t have a current roadmap for response pre-filling, but we will keep this in mind! For the DPO datasets, this can be obtained through human annotation, or some kind of A/B testing flow. For synthetic data generation, you can also explore some kind of rejection sampling with a model and an evaluation to help generate preferred and non-preferred outputs from the same prompt.

Question:

@Ryan121

Have you tried generating your own datasets for DPO using O1?

@andrewpeng

Personally, I haven’t - but this would be an interesting approach to try if you have a use case in mind!

Question:

@christiano

When can we expect the voices from advanced mode will be made available to the API. Right now they are two different voice. And when will you make available tools to control the voice emotion, tone, innotation, etc?

@jeffsharris

No current plans to bring the ChatGPT voices to API

In terms of controlling the voices, we’ve done a ton to make them steerable., especially on today’s new models. Definitely try prompting them to speak the way you want (accent, speed, excitement, age, emotion, etc…)

Question:

@356

Here’s a clearer version of your question:

Would it be possible to automate the DPO (Direct Preference Optimization) training data collection process? I’m considering two approaches:

1. Using a fine-tuned model as a preference judge
2. Using an OpenAI model as a general preference judge

The workflow would be:

• Input: List of prompts
• Output: Multiple model responses per prompt
• Judge model: Assigns preferences between responses
• Result: Automated DPO training dataset

Additionally, can DPO handle multi-turn conversations where we provide a complete dialogue (user-assistant-user-assistant) with only the final assistant response varying?

@andrewpeng

Yes, it is definitely possible to do synthetic data generation for DPO, an important thing to keep in mind is to make sure your evaluator for generated samples is consistent across data points. Also, it is possible to fine-tune for multi-turn conversations! Just put the final assistant message in the preferred / non-preferred output.

Question:

@morg

Just FYI the python/JS code blocks in the docs here are reversed

https://platform.openai.com/docs/guides/reasoning

@romainhuet

Thank you for the catch! We’ll fix this right away.

Question:

@misreadable_mind

Thanks for the GO and Java SDK’s. Any updates for Official Open AI PHP SDK?

@romainhuet

No timeline yet for a PHP SDK, but we’re excited to ultimately support all the most popular languages! We’ll keep you updated, and in the meantime, we recommend community-supported libraries to get started.

Question:

@chinmay1

Is Realtime API out of BETA now?

@jeffsharris

It’s still in Beta. There are a few more goodies we’d like to add before declaring it a GA

Question:

@harperg

Is there a plan to have fine tuning for the audio models? And if so, when can we expect it?

@jeffsharris

It’s something we want, but nothing to share just yet. Our biggest focus at the moment is making sure the base audio models are very high quality

Question:

@morg

Great mini dev day!

Would love more clarity on using System Message vs Developer Message in the api? Has System Message gone away? Or Developer Messages are an addition to it?

@michellep

Developer messages are new in o1 as part of our recent work on instruction hierarchy. You can read more about them in the model spec. You don’t have to think too much about the difference between these two, it’s effectively a no op for devs but we wanted to let you know in case any of your messages reference “system messages” so you can update it to reference “developer messages” instead.

Question:

@sashirestela

Could you please update the list of API Community Libraries? There are some that are outdated and archived.

For example, in Java, the listed library was abandoned months ago by its author, and on the other hand, there are more up-to-date alternatives such as: simple-openai (I’m the author) or Lambdua/openai4j to name a few.

By the way, I followed your procedure to add my library to the list several times, but it never worked and I never received any feedback.

@kwhinnery

Thanks for calling this out - I will both make this change, and revise the process for sharing these updates.

Question:

@lukeduncan333

When will we be able to fine-tune voice models?
Is this something in the works or on the roadmap?

@andrewpeng

We have heard good feedback regarding audio / multi-modal finetuning, but right now we don’t have plans to release audio fine-tuning soon.

Question:

@abradjan

Thanks for this AMA! Is there any word on when assistants code interpreter tool will be out of preview/beta?

@romainhuet

We plan to improve the Assistants API next year. You can continue to use our code_interpreter tool in your apps and we’ll continue to ensure its performance!

Question:

@tomfuller71

Assistants api has been in beta with associated rate limits for over a year now - why should we as developers continue to waste time prototyping solutions with OpenAI when we can’t actually ship anything. No roadmap or communication. Either kill it or release it - permanent betas are just toxic.

@nikunj

I understand this has taken longer than expected but we plan to iterate on and improve the Assistants API next year. In the meantime, we’re committed to making sure that your apps on the Assistants API work performantly.

Question:

@kimm

Assistant API: citations with quoted text and bounding boxes of the quoted text.

@nikunj

Sounds like a cool idea – we’ll discuss it with the team!

Question:

@michael.m.dowling

Can we expect visual grounding for references cited by Assistants API, or visual grounding more generally?

@nikunj

Not immediately, but this is an interesting idea!

Question:

@zavocc

Is caching available in the assistants api?

@nikunj

Yes, it is available in Assistants

Question:

@codebird

1. Can the Vector Stores associated with Assistants API be used by third party LLMs like OSS models/Gemini for context retrieval? Or it has to be one of the OpenAI models only?
2. Is there a limit on the creation of number of Assistants? And can each Assistant along with the associated Vector Stores be associated with a unique organization, so that each organization has a separate RAG store of its own segmented by Assistant ID?

Wish, you people brought some improvements to the Assistant API as well today.

@nikunj

• Vector stores can’t be used outside the Assistants API at the moment, but we’ll take this as a feature request!
• No limits on number of Assistants.

Question:

@anisingh2000

Hey team!!
Congrats on all the releases
Just curious, are there any plans to allow PDF input for the Chat Completion API? Claude added it to their API for sonnet 3.5 and allowing PDF upload and it’s much better than having the developer handle the PDF to image conversion and sending it as images.

@nikunj

This is a really useful feature and we’ll work on bringing this to the OpenAI API.

Question:

@x1f4r

If I can control how long/hard o1 thinks in the API, what’s the difference with o1-pro (the upcoming API version), which is designed to think longer/harder?

@nikunj

A high reasoning_effort tells the model to think longer in a single model turn.

o1-pro uses techniques that go beyond thinking for longer and we’re planning on supporting o1 pro mode in the API!

Question:

@chinmay1

Does the new WebRTC requires (for Realtime voice) VAD implement notation or is that handled by OpenAI backend?

@juberti

VAD functionality is automatically handled when using the WebRTC API.

Question:

@danizord

what’s the difference between o1-pro and o1 with long reasoning_effort?

@nikunj

o1-pro uses techniques that go beyond thinking for longer. We’re working on bringing o1 pro mode to the API soon!

Question:

@SomeUser2022

More generally, what are we as developers not doing as much as you think we should? What do you wish we did differently, or more or less of? We take constructive criticism too

@michellep

One big one is evals! I see tons of developers not using evals at all and relying on vibes for rolling out changes to prod. Would highly recommend creating some simple evals using our evals product (or open source offerings) so you can update with confidence when we release new models.

[platypus]

This is awesome, thank you @VeitB and all OpenAI staff who took time to answer these great Qs.

[sashirestela]

Hey OpenAI Staff guys, very thanks for your time to answer most of our questions. We hope you could take them to improve the OpenAI API.!!

Sashir

[Foxalabs]

Awesome work collating all of that, VB!