Just watched this video: https://www.youtube.com/watch?v=8Zpwi51xxNk&ab_channel=WesRoth
20x price reduction? Now, that would really “change everything”. If true.
1 Like
It says the 20x is due to OpenAI including “memory storage”. So it’s not saying the API cost is going down, just the addition of memory is going to make the overall “AI price”, which includes memory, goes down.
I’m assuming this is pointed at the embedding crowd that uses expensive per instance vector databases, instead of running their own database, even in the public cloud, at a huge discount.
We’ll see what materializes, but my guess they will just release the summary, similar to what is in ChatGPT, in the API version, so that you don’t need to rely on vector databases so easily.
It would be pretty easy for them to do this, but I think it would require some sort of UUID identifying the exact conversation, so it doesn’t mix multiple conversations together.
Just speculation, but would help a lot of struggling folks in the chat API space, without requiring expensive, or time consuming setups to get basic chat flow working.
3 Likes
I’m in that crowd! I could probably save by bringing it in house, but I’m sort of happy with the way things have been working so far – at least up until today!
1 Like
Don’t get your news from youtube talking faces.
from ‘unnamed sources’
The planned release of the so-called stateful API (Application Program Interface) will make it cheaper for companies to create applications by remembering the conversation history of inquiries. This could dramatically reduce the amount of usage developers need to pay for. Currently, processing a one-page document using GPT-4 could cost 10 cents, depending on the length and complexity of the input and output, according to pricing on OpenAI’s website.
This seems to be imaginings. Inference cost is what it is, the time monopolizing an AI instance for generation. But the the computation we are getting per token for the same price as March is clearly reduced.
Yes, a TON of people are in that crowd. It will help you, and that’s good! This is why it makes sense. It’s what I see all day, so it should be obvious to OpenAI at this point, we’ll see 
Agree, but the value isn’t 100% inference. If you are a business owner, and want conversational flow and recall, you need memory, and memory is an add-on that requires developers (not cheap) or time (not free), so huge value is gained outside of the inference box by bringing these externals to the community for “free”, and without much effort on OpenAI’s part.
This add-on, without decreasing inference cost, would decrease the overall system cost of operating, and would likely be ~20x. (not paying people, not paying clouds, and not paying expensive vector databases!)
Unnamed sources aren’t automatically unreliable. Reuters has a very strong reputation for quality journalism.
Absolutely true!
That said, if they wanted to cut API cost per token by 80% across the board I wouldn’t be upset.
2 Likes
Due to competitive market forces, you can bet API costs will be cut 80% someday. 
Market leaders always can charge a premium as long as they are market leaders and the demand appreciates them as such
2 Likes
Wondering if ChatGPT plug-ins are on the way out if it will be acceptable in the future to scrape and manipulate the web-ui without running into issues with bot detection.
I know some browser extensions do this successfully but it would be somewhat a relief to know my extensions won’t cause harm to the users.
Thanks for sharing, honestly a bit disappointed in Reuters, their “unnamed source” could be a random guy they found on the street, or it could be Altman himself.
Assuming it’s correct, I’ll say it sounds like they will just be allowing developers to use the same content management system they’re using for ChatGPT.
1 Like
Just FYI, Reuters is a very trustworthy news organization. Their unnamed source almost certainly works for OpenAI and is in a position to know, or they wouldn’t have gone to press with what they have.
You’ll also note the article cites unnamed sources, so they have more than one person who is in a position to know sharing this information.
So, while it could be wrong or OpenAI could even change plans in the next month, it’s wholly without merit to accuse the journalists of fabricating the story.
2 Likes
Indeed, that’s why it stood out to me, I’m not here to accuse them of fabricating the story, I was more focussed on the semantics. Later in the article we get a bit more information:
said sources, who asked not to be named for discussing the company’s private plans.
Which makes it sound like an employee, but it could also be a 3rd party who’s associated with planning the event. I don’t know, but I think they do, I’m just asking the question “is this a verified source?”