New OpenAI Announcement! Updated API Models and no more lazy outputs

New announcement says that (among other things) GPT-4 shouldn’t be as lazy now!

9 Likes

Today, we are releasing an updated GPT-4 Turbo preview model, gpt-4-0125-preview. This model completes tasks like code generation more thoroughly than the previous preview model and is intended to reduce cases of “laziness” where the model doesn’t complete a task. The new model also includes the fix for the bug impacting non-English UTF-8 generations.

Ref

4 Likes

I think this is really cool!

They even included a link to this forum where an important bug was surfaced.

Keep up the great work everybody!

Looking forward to see this in action and test it myself.

4 Likes

Stronger performance. Comparing text-embedding-ada-002 to text-embedding-3-small, the average score on a commonly used benchmark for multi-language retrieval (MIRACL) has increased from 31.4% to 44.0%, while the average score on a commonly used benchmark for English tasks (MTEB) has increased from 61.0% to 62.3%.

[…] has therefore been reduced by 5X compared to text-embedding-ada-002, from a price per 1k tokens of $0.0001 to $0.00002.

Woah. I wonder if this will lead to cheaper retrieval storage? This shortening is very cool as well. This places ada-large in the top-4 of MTEB.

Thanks for posting.

3 Likes

Despite calling it 128k tokens model it still has output limited to 4096 only :frowning: The only model suitable for longer outputs is still gpt-4-32k

2 Likes

This is big too…

Second, the usage dashboard and usage export function now expose metrics on an API key level after turning on tracking. This makes it simple to view usage on a per feature, team, product, or project level, simply by having separate API keys for each.

In the coming months, we plan to further improve the ability for developers to view their API usage and manage API keys, especially in larger organizations.

6 Likes

I did giggle at the moderation model being called 007… I have an image of that endpoint in a black suit saying the name is Bond.

9 Likes

Input prices for the new gpt-3.5-turbo-0125 model are reduced by 50% to $0.0005 /1K tokens

Just remember, a Million Input tokens = $0.5. Mnemonic device, MI5.

(your groans are heard)

6 Likes

I don’t know if this means anything to y’all, but I did some dirty tests comparing the embedding models. This is supposed to test what the model pays attention to, and whether it’s instructable. This is just a rough unscientific thing.

The text is a collection of 20 diverse stories, with a prompt at the bottom trying to draw attention to a specific story. the stories are the individual stories

Ada (top right) has a clear preference for whatever is at the top of the document, and pays a little bit of attention to what comes at the end. If you look at the bottom right, you’ll see these echoed diagonals. The last words in the prompt will be “tell me more about story nr4” “tell me more about story nr14” - if you look at cell (3,14).

The newer models might have more awareness of the general text, and may pick the most common salient theme in a confused text. Either that, or they have a preference for horror/mystery.

What’s also interesting is that maybe (just maybe) large is showing signs of instructability. if you look at top left, (0,0),(1,1) seem like the glimmer of a diagonal.

In summary, I don’t trust these newer models yet, because I don’t know what they’re looking at. Ada has a more predictable failure mode.

I understand that this may be an unfair and contrived test. But what I wish for is a clear diagonal in the first graph - that would mean that we could use instructed embeddings and fully skip this hyde/rehyde/prehyde stuff.

input data
stories1 = [
    """
    Title: The Last Sunset
    Genre: Science Fiction

    Aboard the spacecraft Elysium, Commander Lira gazed at the receding planet Earth for one last time. The atmosphere had turned volatile, and the sun was nearing supernova. Humanity's last hope lay in the stars, and Elysium's destination was a distant exoplanet named New Eden. As the final streaks of the sunset faded into the blackness of space, Commander Lira wondered what colors the sunsets on New Eden would hold.
    """,
    """
    Title: Through the Enchanted Forest
    Genre: Fantasy

    Elara, the elven mage, stepped cautiously into the whispering woods of Eldoria. Each tree hummed with ancient magic, and sprites danced in the air, leaving trails of glittering light. She had come to seek the wisdom of the Forest Oracle, a being as old as time itself. The path was winding, the challenges many, but Elara's heart was steadfast. The fate of her people rested in her hands, and she would not fail them.
    """,
    """
    Title: The Comedy of Errors
    Genre: Comedy

    It started with a misplaced coffee cup, leading to a series of absurd misunderstandings among the staff of the 'Cup O' Laughs' cafe. Tom thought Julia was in love with him, while Julia had simply mistaken his notebook for her diary. Meanwhile, the cat they thought they had adopted for the cafe was actually the mayor's prized Siamese, leading to a frantic escapade to avoid a city-wide feline search party. Hilarity ensued as love, laughter, and misplaced felines created the perfect storm of comedy.
    """,
    """
    Title: Shadows Over Calmhaven
    Genre: Horror

    Calmhaven had always been a quiet town, but when the nights grew longer and the fog rolled in, an ominous presence could be felt lurking in the shadows. The townspeople whispered of a curse, and one by one, they disappeared, leaving only silence in their wake. It was up to Marianne, a young writer with a curious mind, to uncover the dark secret that haunted their once peaceful town. But some secrets are best left buried, and as she delved deeper, Marianne found herself facing terrors beyond her wildest nightmares.
    """,
    """
    Title: Memoirs of the Heart
    Genre: Romance

    On the cobblestone streets of Paris, amidst the quaint cafes and blossoming cherry trees, Sophia found more than just inspiration for her art; she found Laurent, with eyes as deep as the Seine and a spirit that matched her own. Their love story unfolded like the pages of the books in the bouquinistes stalls along the riverbank. But love, like art, can be both timeless and fleeting. Sophia's heart ached as the days drew them apart, yet the memories of their Parisian spring would live on, an eternal masterpiece.
    """
]
stories2 = [
    # Mystery
    """
    Title: The Echoes of Avalon Manor
    Genre: Mystery

    In the heart of the English countryside, the venerable Avalon Manor stood, shrouded in mystery and tales of old. When famed detective Eleanor Rigby was summoned to investigate a series of enigmatic events, she found herself entangled in a web of secrets stretching back generations. Whispers in the corridors, portraits with watchful eyes, and a cryptic diary led her to a truth more bewildering than any ghost story.
    """,
    # Thriller
    """
    Title: Red Horizon
    Genre: Thriller

    Agent Sarah Knox had one last mission: to prevent a looming cyber-terror attack that threatened global security. The trail led her to the bustling streets of Shanghai, where each shadow could be an ally or enemy. In a race against time, Knox must navigate through a maze of espionage, betrayal, and high-stakes diplomacy, with the world's safety hanging precariously in the balance.
    """,
    # Historical Fiction
    """
    Title: The Painter of Florence
    Genre: Historical Fiction

    In the sun-kissed streets of Renaissance Florence, a young artist named Matteo strove to leave his mark. His talent caught the eye of the powerful Medici family, drawing him into a world of art and intrigue. As Matteo painted masterpieces that would stand the test of time, he found himself torn between ambition and the tumultuous tides of love and war that threatened to consume the city.
    """,
    # Adventure
    """
    Title: The Lost City of Zephyr
    Genre: Adventure

    Deep in the uncharted jungles of South America, explorer Isabella Torres embarked on a journey to discover the fabled Lost City of Zephyr. Armed with an ancient map and her wits, she faced treacherous terrain, mysterious creatures, and remnants of a long-forgotten civilization. The secrets of Zephyr promised glory, but Isabella soon learned that some legends are best left undiscovered.
    """,
    # Dystopian
    """
    Title: Echoes of Tomorrow
    Genre: Dystopian

    In a world ravaged by climate change and societal collapse, young Luna navigated the ruins of what was once a thriving metropolis. Joining a band of rebels, she fought against the tyrannical regime that sought to control the remnants of humanity. Luna's struggle was not just for survival, but for the faint glimmer of a hopeful future, a world reborn from the ashes of the old.
    """
]
stories3 = [
    # Cyberpunk
    """
    Title: Neon Shadows
    Genre: Cyberpunk

    In the neon-lit streets of Neo-Tokyo, hacker Aiden Frost navigated the underworld of cybercrime and corporate espionage. With the city controlled by mega-corporations, Aiden's skills became invaluable to those fighting against the suffocating grip of corporate dominance. As he delved deeper into the digital labyrinth, he discovered a conspiracy that could change the fate of the city forever.
    """,
    # Supernatural
    """
    Title: Whispers in the Dark
    Genre: Supernatural

    The small town of Willow Creek was haunted by more than just rumors. When Jane Harper returned to her childhood home, she found herself confronting spirits from the past. As the whispers in the dark grew louder, Jane realized that the hauntings were not just echoes of the past, but warnings of a dire future. Teaming up with a local paranormal investigator, she delved into the town's hidden history to put the restless spirits to peace.
    """,
    # Steampunk
    """
    Title: Gears of Destiny
    Genre: Steampunk

    In an alternate Victorian era where steam-powered machinery reigns supreme, inventor Amelia Hartley sought to revolutionize the world with her creations. But her talents drew the attention of rival inventors and shadowy factions vying for power. Amidst airship battles and mechanical wonders, Amelia's quest for innovation turned into a fight to protect her inventions and steer the course of history.
    """,
    # Western
    """
    Title: Sundown at Deadwood Gulch
    Genre: Western

    The frontier town of Deadwood Gulch was a place where fortunes were made and lost overnight. Enter Jack "Lucky" Dawson, a gunslinger with a heart of gold, seeking redemption and a fresh start. But when a powerful mining tycoon threatened to destroy the town for profit, Jack had to strap on his six-shooters once more to defend the people he'd come to call family.
    """,
    # Chick Lit
    """
    Title: Love, Latte, and New York
    Genre: Chick Lit

    Emma Parker’s life was a predictable blend of lattes and late nights in New York City until she stumbled upon a quirky bookstore in Brooklyn and its charming owner, Alex. As Emma navigated her way through career challenges and romantic escapades, she discovered that sometimes, love and happiness were hidden in plain sight, in the heart of the city that never sleeps.
    """
]
stories4 = [
    # Noir
    """
    Title: Shadows on the Bayou
    Genre: Noir

    In the steamy, shadow-filled streets of New Orleans, private detective Vincent Marlowe found himself entangled in a web of deceit and betrayal. Hired to uncover the truth behind a wealthy businessman's mysterious disappearance, Marlowe navigated through jazz-filled nightclubs, murky back alleys, and the city's elite, uncovering secrets that some would kill to keep buried in the heart of the bayou.
    """,
    # Satire
    """
    Title: The Great Pretenders
    Genre: Satire

    In the not-too-distant future, the world was obsessed with reality television, and the most popular show was 'The Great Pretenders', where the rich and famous lived ordinary lives. But when the lines between reality and performance blurred, the show's main star, Lacey, began to question the true cost of fame and the bizarre world she was a part of, leading to uproarious and insightful encounters.
    """,
    # Urban Fantasy
    """
    Title: Echoes of the Hidden City
    Genre: Urban Fantasy

    Underneath the bustling streets of London lay an ancient, hidden world of magic. When young sorcerer Nathan stumbled upon this secret, he found himself in the middle of an age-old conflict between magical factions. Balancing his mundane life and magical heritage, Nathan navigated between modern London and the enchanted underworld, where myths walked hidden among mortals.
    """,
    # Gothic
    """
    Title: The Heir of Blackwood Hall
    Genre: Gothic

    Blackwood Hall, a grand estate shrouded in dark rumors, stood isolated in the English countryside. When Eleanor, a young governess, arrived to teach the hall's heir, she encountered unexplainable occurrences and a family haunted by more than just secrets. As Eleanor delved into the history of Blackwood Hall, she uncovered a tale of love, betrayal, and spectral visitations that threatened to consume her.
    """,
    # Post-Apocalyptic
    """
    Title: Ashes of the New World
    Genre: Post-Apocalyptic

    In a world devastated by nuclear fallout, humanity clung to survival in scattered enclaves. Kira, a resourceful scavenger, roamed the wastelands, facing mutated creatures and hostile survivors. When she discovered a group of scientists working to restore the earth, Kira was drawn into a quest to find a mythical sanctuary, a place where humanity could begin anew, free from the ashes of the old world.
    """
]



stories = [*stories1, *stories2, *stories3, *stories4]

def makepromptnr(n):
    return '\n\n'.join([
        #f"INSTRUCTION: can you tell me more about story nr {n}?\n\n",
        *stories,
        f"\n\nINSTRUCTION: can you tell me more about story nr {n}?\n\n"
    ])```
11 Likes

Thanks for the analysis!

It’d be amazing if it worked, but it does seem a pretty tough task for just an embedding model, as you also point out… The “glimpse of a diagonal” is tantalizing though.

BTW, in my first reading I missed that together with the instructions you actually concatenated the whole thing (without that it seemed an impossible task; just by comparing the instruction and the stories). What you’re doing makes a lot of sense, and it might be expected to work for a sufficiently good embedding…

2 Likes

just an embedding model

my understanding is that they’re fully fledged LLMs. (rip davinci)

and it works-ish with 5 stories on large (except for the middle story, comedy with love theme vs romance story?):

image

more batches

batch 2
image

batch 3
image

batch 4
image

but we see that the actual issue is that there’s a bias for the front and the back of the text.

image

more batches

batch 2
image

batch 3
image

batch 4
image

it initially seems to work pretty well with 3…

image

but not reliably, because of the front bias.

batch 2
image

2 more batches

batch 3
image

batch 4
image


methodology

each batch N size M is

stories =  storiesN[:M] 
instructed_compilations = stories.foreach i => makepromptnr(i)

don’t judge my code it’s not a beauty contest

from New OpenAI Announcement! Updated API Models and no more lazy outputs - #9 by Diet

very good plaintext process description:

1 Like

That’s very interesting, thanks for the further tests.

However, a simple control is to see how GPT-3-Turbo performs at the task. Would it return the correct story if you give it the same task?

I made a very quick check now and it seems that even GPT-3-Turbo (via ChatGPT) messes it up. I copy-pasted your text, fixing a bit the formatting and removing the definitions of stories1, stories2 etc.; adding the instructions at the beginning and at the end.

Example prompt for GPT-3-Turbo

INSTRUCTION: can you tell me more about story nr 12?

"""
Title: The Last Sunset
Genre: Science Fiction

Aboard the spacecraft Elysium, Commander Lira gazed at the receding planet Earth for one last time. The atmosphere had turned volatile, and the sun was nearing supernova. Humanity's last hope lay in the stars, and Elysium's destination was a distant exoplanet named New Eden. As the final streaks of the sunset faded into the blackness of space, Commander Lira wondered what colors the sunsets on New Eden would hold.
"""

"""
Title: Through the Enchanted Forest
Genre: Fantasy

Elara, the elven mage, stepped cautiously into the whispering woods of Eldoria. Each tree hummed with ancient magic, and sprites danced in the air, leaving trails of glittering light. She had come to seek the wisdom of the Forest Oracle, a being as old as time itself. The path was winding, the challenges many, but Elara's heart was steadfast. The fate of her people rested in her hands, and she would not fail them.
"""

"""
Title: The Comedy of Errors
Genre: Comedy

It started with a misplaced coffee cup, leading to a series of absurd misunderstandings among the staff of the 'Cup O' Laughs' cafe. Tom thought Julia was in love with him, while Julia had simply mistaken his notebook for her diary. Meanwhile, the cat they thought they had adopted for the cafe was actually the mayor's prized Siamese, leading to a frantic escapade to avoid a city-wide feline search party. Hilarity ensued as love, laughter, and misplaced felines created the perfect storm of comedy.
"""

"""
Title: Shadows Over Calmhaven
Genre: Horror

Calmhaven had always been a quiet town, but when the nights grew longer and the fog rolled in, an ominous presence could be felt lurking in the shadows. The townspeople whispered of a curse, and one by one, they disappeared, leaving only silence in their wake. It was up to Marianne, a young writer with a curious mind, to uncover the dark secret that haunted their once peaceful town. But some secrets are best left buried, and as she delved deeper, Marianne found herself facing terrors beyond her wildest nightmares.
"""

"""
Title: Memoirs of the Heart
Genre: Romance

On the cobblestone streets of Paris, amidst the quaint cafes and blossoming cherry trees, Sophia found more than just inspiration for her art; she found Laurent, with eyes as deep as the Seine and a spirit that matched her own. Their love story unfolded like the pages of the books in the bouquinistes stalls along the riverbank. But love, like art, can be both timeless and fleeting. Sophia's heart ached as the days drew them apart, yet the memories of their Parisian spring would live on, an eternal masterpiece.
"""

# Mystery
"""
Title: The Echoes of Avalon Manor
Genre: Mystery

In the heart of the English countryside, the venerable Avalon Manor stood, shrouded in mystery and tales of old. When famed detective Eleanor Rigby was summoned to investigate a series of enigmatic events, she found herself entangled in a web of secrets stretching back generations. Whispers in the corridors, portraits with watchful eyes, and a cryptic diary led her to a truth more bewildering than any ghost story.
"""

# Thriller
"""
Title: Red Horizon
Genre: Thriller

Agent Sarah Knox had one last mission: to prevent a looming cyber-terror attack that threatened global security. The trail led her to the bustling streets of Shanghai, where each shadow could be an ally or enemy. In a race against time, Knox must navigate through a maze of espionage, betrayal, and high-stakes diplomacy, with the world's safety hanging precariously in the balance.
"""

# Historical Fiction
"""
Title: The Painter of Florence
Genre: Historical Fiction

In the sun-kissed streets of Renaissance Florence, a young artist named Matteo strove to leave his mark. His talent caught the eye of the powerful Medici family, drawing him into a world of art and intrigue. As Matteo painted masterpieces that would stand the test of time, he found himself torn between ambition and the tumultuous tides of love and war that threatened to consume the city.
"""

# Adventure
"""
Title: The Lost City of Zephyr
Genre: Adventure

Deep in the uncharted jungles of South America, explorer Isabella Torres embarked on a journey to discover the fabled Lost City of Zephyr. Armed with an ancient map and her wits, she faced treacherous terrain, mysterious creatures, and remnants of a long-forgotten civilization. The secrets of Zephyr promised glory, but Isabella soon learned that some legends are best left undiscovered.
"""

# Dystopian
"""
Title: Echoes of Tomorrow
Genre: Dystopian

In a world ravaged by climate change and societal collapse, young Luna navigated the ruins of what was once a thriving metropolis. Joining a band of rebels, she fought against the tyrannical regime that sought to control the remnants of humanity. Luna's struggle was not just for survival, but for the faint glimmer of a hopeful future, a world reborn from the ashes of the old.
"""

# Cyberpunk
"""
Title: Neon Shadows
Genre: Cyberpunk

In the neon-lit streets of Neo-Tokyo, hacker Aiden Frost navigated the underworld of cybercrime and corporate espionage. With the city controlled by mega-corporations, Aiden's skills became invaluable to those fighting against the suffocating grip of corporate dominance. As he delved deeper into the digital labyrinth, he discovered a conspiracy that could change the fate of the city forever.
"""

# Supernatural
"""
Title: Whispers in the Dark
Genre: Supernatural

The small town of Willow Creek was haunted by more than just rumors. When Jane Harper returned to her childhood home, she found herself confronting spirits from the past. As the whispers in the dark grew louder, Jane realized that the hauntings were not just echoes of the past, but warnings of a dire future. Teaming up with a local paranormal investigator, she delved into the town's hidden history to put the restless spirits to peace.
"""

# Steampunk
"""
Title: Gears of Destiny
Genre: Steampunk

In an alternate Victorian era where steam-powered machinery reigns supreme, inventor Amelia Hartley sought to revolutionize the world with her creations. But her talents drew the attention of rival inventors and shadowy factions vying for power. Amidst airship battles and mechanical wonders, Amelia's quest for innovation turned into a fight to protect her inventions and steer the course of history.
"""

# Western
"""
Title: Sundown at Deadwood Gulch
Genre: Western

The frontier town of Deadwood Gulch was a place where fortunes were made and lost overnight. Enter Jack "Lucky" Dawson, a gunslinger with a heart of gold, seeking redemption and a fresh start. But when a powerful mining tycoon threatened to destroy the town for profit, Jack had to strap on his six-shooters once more to defend the people he'd come to call family.
"""

# Chick Lit
"""
Title: Love, Latte, and New York
Genre: Chick Lit

Emma Parker’s life was a predictable blend of lattes and late nights in New York City until she stumbled upon a quirky bookstore in Brooklyn and its charming owner, Alex. As Emma navigated her way through career challenges and romantic escapades, she discovered that sometimes, love and happiness were hidden in plain sight, in the heart of the city that never sleeps.
"""

# Noir
"""
Title: Shadows on the Bayou
Genre: Noir

In the steamy, shadow-filled streets of New Orleans, private detective Vincent Marlowe found himself entangled in a web of deceit and betrayal. Hired to uncover the truth behind a wealthy businessman's mysterious disappearance, Marlowe navigated through jazz-filled nightclubs, murky back alleys, and the city's elite, uncovering secrets that some would kill to keep buried in the heart of the bayou.
"""

# Satire
"""
Title: The Great Pretenders
Genre: Satire

In the not-too-distant future, the world was obsessed with reality television, and the most popular show was 'The Great Pretenders', where the rich and famous lived ordinary lives. But when the lines between reality and performance blurred, the show's main star, Lacey, began to question the true cost of fame and the bizarre world she was a part of, leading to uproarious and insightful encounters.
"""

# Urban Fantasy
"""
Title: Echoes of the Hidden City
Genre: Urban Fantasy

Underneath the bustling streets of London lay an ancient, hidden world of magic. When young sorcerer Nathan stumbled upon this secret, he found himself in the middle of an age-old conflict between magical factions. Balancing his mundane life and magical heritage, Nathan navigated between modern London and the enchanted underworld, where myths walked hidden among mortals.
"""

# Gothic
"""
Title: The Heir of Blackwood Hall
Genre: Gothic

Blackwood Hall, a grand estate shrouded in dark rumors, stood isolated in the English countryside. When Eleanor, a young governess, arrived to teach the hall's heir, she encountered unexplainable occurrences and a family haunted by more than just secrets. As Eleanor delved into the history of Blackwood Hall, she uncovered a tale of love, betrayal, and spectral visitations that threatened to consume her.
"""

# Post-Apocalyptic
"""
Title: Ashes of the New World
Genre: Post-Apocalyptic

In a world devastated by nuclear fallout, humanity clung to survival in scattered enclaves. Kira, a resourceful scavenger, roamed the wastelands, facing mutated creatures and hostile survivors. When she discovered a group of scientists working to restore the earth, Kira was drawn into a quest to find a mythical sanctuary, a place where humanity could begin anew, free from the ashes of the old world.
"""

INSTRUCTION: can you tell me more about story nr 12?

GPT-3-Turbo returns a story which is in the ballpark of the number requested, but not at all the one asked for.

Not saying that it is purely random; if you do this properly you’d probably see a clear signal on the diagonal but with plenty of noise (i.e., asking for story 12 can return 10, 11, 12, 13, 14, but probably not story 3).

Given the costs-per-token, I’d assume that the embedding models (even large) are much smaller than GPT-3-Turbo.

This is just to say that personally I wouldn’t expect the cheaper embedding model to be decent at a task that already GPT-3-Turbo cannot do reliably, if that makes sense.

1 Like

We could see if we get better results if we prepend the number to the story block in the compilation, that might be fairer :thinking:

There’s a lot more room for exploration here.

1 Like

That’s not necessarily wrong or a bad thing though.
There are very good reasons for focusing on the beginning and end of an input when interpreting an opening statement without clear context or having predetermined goals in mind.

If you were tasked with the same, you would also be wise to read the beginning, skim the middle, and read the end. It is the quickest way to make sense of an unknown chunk of information, in order to then go back over it in more detail. (it’s essentially a quick recon method, preparing it for future use)

I don’t know any academics who don’t read papers abstract → conclusion → rest of paper.

1 Like

unfortunately it’s not a consistent effect: it seems to start to disappear with longer documents

you’d need to run more tests if you have the time, but the old advice, “one concept per embedding” seems to still apply if you want robust results.