I don’t know if this means anything to y’all, but I did some dirty tests comparing the embedding models. This is supposed to test what the model pays attention to, and whether it’s instructable. This is just a rough unscientific thing.
The text is a collection of 20 diverse stories, with a prompt at the bottom trying to draw attention to a specific story. the stories are the individual stories
Ada (top right) has a clear preference for whatever is at the top of the document, and pays a little bit of attention to what comes at the end. If you look at the bottom right, you’ll see these echoed diagonals. The last words in the prompt will be “tell me more about story nr4” “tell me more about story nr14” - if you look at cell (3,14).
The newer models might have more awareness of the general text, and may pick the most common salient theme in a confused text. Either that, or they have a preference for horror/mystery.
What’s also interesting is that maybe (just maybe) large is showing signs of instructability. if you look at top left, (0,0),(1,1) seem like the glimmer of a diagonal.
In summary, I don’t trust these newer models yet, because I don’t know what they’re looking at. Ada has a more predictable failure mode.
I understand that this may be an unfair and contrived test. But what I wish for is a clear diagonal in the first graph - that would mean that we could use instructed embeddings and fully skip this hyde/rehyde/prehyde stuff.
input data
stories1 = [
"""
Title: The Last Sunset
Genre: Science Fiction
Aboard the spacecraft Elysium, Commander Lira gazed at the receding planet Earth for one last time. The atmosphere had turned volatile, and the sun was nearing supernova. Humanity's last hope lay in the stars, and Elysium's destination was a distant exoplanet named New Eden. As the final streaks of the sunset faded into the blackness of space, Commander Lira wondered what colors the sunsets on New Eden would hold.
""",
"""
Title: Through the Enchanted Forest
Genre: Fantasy
Elara, the elven mage, stepped cautiously into the whispering woods of Eldoria. Each tree hummed with ancient magic, and sprites danced in the air, leaving trails of glittering light. She had come to seek the wisdom of the Forest Oracle, a being as old as time itself. The path was winding, the challenges many, but Elara's heart was steadfast. The fate of her people rested in her hands, and she would not fail them.
""",
"""
Title: The Comedy of Errors
Genre: Comedy
It started with a misplaced coffee cup, leading to a series of absurd misunderstandings among the staff of the 'Cup O' Laughs' cafe. Tom thought Julia was in love with him, while Julia had simply mistaken his notebook for her diary. Meanwhile, the cat they thought they had adopted for the cafe was actually the mayor's prized Siamese, leading to a frantic escapade to avoid a city-wide feline search party. Hilarity ensued as love, laughter, and misplaced felines created the perfect storm of comedy.
""",
"""
Title: Shadows Over Calmhaven
Genre: Horror
Calmhaven had always been a quiet town, but when the nights grew longer and the fog rolled in, an ominous presence could be felt lurking in the shadows. The townspeople whispered of a curse, and one by one, they disappeared, leaving only silence in their wake. It was up to Marianne, a young writer with a curious mind, to uncover the dark secret that haunted their once peaceful town. But some secrets are best left buried, and as she delved deeper, Marianne found herself facing terrors beyond her wildest nightmares.
""",
"""
Title: Memoirs of the Heart
Genre: Romance
On the cobblestone streets of Paris, amidst the quaint cafes and blossoming cherry trees, Sophia found more than just inspiration for her art; she found Laurent, with eyes as deep as the Seine and a spirit that matched her own. Their love story unfolded like the pages of the books in the bouquinistes stalls along the riverbank. But love, like art, can be both timeless and fleeting. Sophia's heart ached as the days drew them apart, yet the memories of their Parisian spring would live on, an eternal masterpiece.
"""
]
stories2 = [
# Mystery
"""
Title: The Echoes of Avalon Manor
Genre: Mystery
In the heart of the English countryside, the venerable Avalon Manor stood, shrouded in mystery and tales of old. When famed detective Eleanor Rigby was summoned to investigate a series of enigmatic events, she found herself entangled in a web of secrets stretching back generations. Whispers in the corridors, portraits with watchful eyes, and a cryptic diary led her to a truth more bewildering than any ghost story.
""",
# Thriller
"""
Title: Red Horizon
Genre: Thriller
Agent Sarah Knox had one last mission: to prevent a looming cyber-terror attack that threatened global security. The trail led her to the bustling streets of Shanghai, where each shadow could be an ally or enemy. In a race against time, Knox must navigate through a maze of espionage, betrayal, and high-stakes diplomacy, with the world's safety hanging precariously in the balance.
""",
# Historical Fiction
"""
Title: The Painter of Florence
Genre: Historical Fiction
In the sun-kissed streets of Renaissance Florence, a young artist named Matteo strove to leave his mark. His talent caught the eye of the powerful Medici family, drawing him into a world of art and intrigue. As Matteo painted masterpieces that would stand the test of time, he found himself torn between ambition and the tumultuous tides of love and war that threatened to consume the city.
""",
# Adventure
"""
Title: The Lost City of Zephyr
Genre: Adventure
Deep in the uncharted jungles of South America, explorer Isabella Torres embarked on a journey to discover the fabled Lost City of Zephyr. Armed with an ancient map and her wits, she faced treacherous terrain, mysterious creatures, and remnants of a long-forgotten civilization. The secrets of Zephyr promised glory, but Isabella soon learned that some legends are best left undiscovered.
""",
# Dystopian
"""
Title: Echoes of Tomorrow
Genre: Dystopian
In a world ravaged by climate change and societal collapse, young Luna navigated the ruins of what was once a thriving metropolis. Joining a band of rebels, she fought against the tyrannical regime that sought to control the remnants of humanity. Luna's struggle was not just for survival, but for the faint glimmer of a hopeful future, a world reborn from the ashes of the old.
"""
]
stories3 = [
# Cyberpunk
"""
Title: Neon Shadows
Genre: Cyberpunk
In the neon-lit streets of Neo-Tokyo, hacker Aiden Frost navigated the underworld of cybercrime and corporate espionage. With the city controlled by mega-corporations, Aiden's skills became invaluable to those fighting against the suffocating grip of corporate dominance. As he delved deeper into the digital labyrinth, he discovered a conspiracy that could change the fate of the city forever.
""",
# Supernatural
"""
Title: Whispers in the Dark
Genre: Supernatural
The small town of Willow Creek was haunted by more than just rumors. When Jane Harper returned to her childhood home, she found herself confronting spirits from the past. As the whispers in the dark grew louder, Jane realized that the hauntings were not just echoes of the past, but warnings of a dire future. Teaming up with a local paranormal investigator, she delved into the town's hidden history to put the restless spirits to peace.
""",
# Steampunk
"""
Title: Gears of Destiny
Genre: Steampunk
In an alternate Victorian era where steam-powered machinery reigns supreme, inventor Amelia Hartley sought to revolutionize the world with her creations. But her talents drew the attention of rival inventors and shadowy factions vying for power. Amidst airship battles and mechanical wonders, Amelia's quest for innovation turned into a fight to protect her inventions and steer the course of history.
""",
# Western
"""
Title: Sundown at Deadwood Gulch
Genre: Western
The frontier town of Deadwood Gulch was a place where fortunes were made and lost overnight. Enter Jack "Lucky" Dawson, a gunslinger with a heart of gold, seeking redemption and a fresh start. But when a powerful mining tycoon threatened to destroy the town for profit, Jack had to strap on his six-shooters once more to defend the people he'd come to call family.
""",
# Chick Lit
"""
Title: Love, Latte, and New York
Genre: Chick Lit
Emma Parker’s life was a predictable blend of lattes and late nights in New York City until she stumbled upon a quirky bookstore in Brooklyn and its charming owner, Alex. As Emma navigated her way through career challenges and romantic escapades, she discovered that sometimes, love and happiness were hidden in plain sight, in the heart of the city that never sleeps.
"""
]
stories4 = [
# Noir
"""
Title: Shadows on the Bayou
Genre: Noir
In the steamy, shadow-filled streets of New Orleans, private detective Vincent Marlowe found himself entangled in a web of deceit and betrayal. Hired to uncover the truth behind a wealthy businessman's mysterious disappearance, Marlowe navigated through jazz-filled nightclubs, murky back alleys, and the city's elite, uncovering secrets that some would kill to keep buried in the heart of the bayou.
""",
# Satire
"""
Title: The Great Pretenders
Genre: Satire
In the not-too-distant future, the world was obsessed with reality television, and the most popular show was 'The Great Pretenders', where the rich and famous lived ordinary lives. But when the lines between reality and performance blurred, the show's main star, Lacey, began to question the true cost of fame and the bizarre world she was a part of, leading to uproarious and insightful encounters.
""",
# Urban Fantasy
"""
Title: Echoes of the Hidden City
Genre: Urban Fantasy
Underneath the bustling streets of London lay an ancient, hidden world of magic. When young sorcerer Nathan stumbled upon this secret, he found himself in the middle of an age-old conflict between magical factions. Balancing his mundane life and magical heritage, Nathan navigated between modern London and the enchanted underworld, where myths walked hidden among mortals.
""",
# Gothic
"""
Title: The Heir of Blackwood Hall
Genre: Gothic
Blackwood Hall, a grand estate shrouded in dark rumors, stood isolated in the English countryside. When Eleanor, a young governess, arrived to teach the hall's heir, she encountered unexplainable occurrences and a family haunted by more than just secrets. As Eleanor delved into the history of Blackwood Hall, she uncovered a tale of love, betrayal, and spectral visitations that threatened to consume her.
""",
# Post-Apocalyptic
"""
Title: Ashes of the New World
Genre: Post-Apocalyptic
In a world devastated by nuclear fallout, humanity clung to survival in scattered enclaves. Kira, a resourceful scavenger, roamed the wastelands, facing mutated creatures and hostile survivors. When she discovered a group of scientists working to restore the earth, Kira was drawn into a quest to find a mythical sanctuary, a place where humanity could begin anew, free from the ashes of the old world.
"""
]
stories = [*stories1, *stories2, *stories3, *stories4]
def makepromptnr(n):
return '\n\n'.join([
#f"INSTRUCTION: can you tell me more about story nr {n}?\n\n",
*stories,
f"\n\nINSTRUCTION: can you tell me more about story nr {n}?\n\n"
])```