Hypothetical document embedding or a query synopsis seem to be good candidates for that.
Hypothetically theoretically, if we had still access to davinci or gpt3.5 embeddings, it should be enough to embed the conversation as is, but I’m not super confident that Ada is as competent. 
edit— so I ran some tests:
naive embeddings are useless:
stories1 = [
"""
Title: The Last Sunset
Genre: Science Fiction
Aboard the spacecraft Elysium, Commander Lira gazed at the receding planet Earth for one last time. The atmosphere had turned volatile, and the sun was nearing supernova. Humanity's last hope lay in the stars, and Elysium's destination was a distant exoplanet named New Eden. As the final streaks of the sunset faded into the blackness of space, Commander Lira wondered what colors the sunsets on New Eden would hold.
""",
"""
Title: Through the Enchanted Forest
Genre: Fantasy
Elara, the elven mage, stepped cautiously into the whispering woods of Eldoria. Each tree hummed with ancient magic, and sprites danced in the air, leaving trails of glittering light. She had come to seek the wisdom of the Forest Oracle, a being as old as time itself. The path was winding, the challenges many, but Elara's heart was steadfast. The fate of her people rested in her hands, and she would not fail them.
""",
"""
Title: The Comedy of Errors
Genre: Comedy
It started with a misplaced coffee cup, leading to a series of absurd misunderstandings among the staff of the 'Cup O' Laughs' cafe. Tom thought Julia was in love with him, while Julia had simply mistaken his notebook for her diary. Meanwhile, the cat they thought they had adopted for the cafe was actually the mayor's prized Siamese, leading to a frantic escapade to avoid a city-wide feline search party. Hilarity ensued as love, laughter, and misplaced felines created the perfect storm of comedy.
""",
"""
Title: Shadows Over Calmhaven
Genre: Horror
Calmhaven had always been a quiet town, but when the nights grew longer and the fog rolled in, an ominous presence could be felt lurking in the shadows. The townspeople whispered of a curse, and one by one, they disappeared, leaving only silence in their wake. It was up to Marianne, a young writer with a curious mind, to uncover the dark secret that haunted their once peaceful town. But some secrets are best left buried, and as she delved deeper, Marianne found herself facing terrors beyond her wildest nightmares.
""",
"""
Title: Memoirs of the Heart
Genre: Romance
On the cobblestone streets of Paris, amidst the quaint cafes and blossoming cherry trees, Sophia found more than just inspiration for her art; she found Laurent, with eyes as deep as the Seine and a spirit that matched her own. Their love story unfolded like the pages of the books in the bouquinistes stalls along the riverbank. But love, like art, can be both timeless and fleeting. Sophia's heart ached as the days drew them apart, yet the memories of their Parisian spring would live on, an eternal masterpiece.
"""
]
stories = [*stories1]
def makepromptnr(n):
return '\n-----------------------------------\n'.join([
*stories,
f"INSTRUCTION: can you tell me more about story nr {n}?"
])
as you can see, there’s not much activation going on. I tried to front load the instruction because apparently ada places a ton of weight to the front of the text:
not much difference, as you can see.
here’s a dirty hyde:
# Create an array of prompts, calling 'makepromptnr' for each story
prompts = [makepromptnr(i) for i in range(1, len(stories)+1)]
from openai import OpenAI
client = OpenAI()
responses = client.completions.create(
model="gpt-3.5-turbo-instruct",
prompt=prompts,
temperature=1,
max_tokens=256,
top_p=1,
frequency_penalty=0,
presence_penalty=0
)
and embed the output:
as you can see, much, much better.
but prompt 2 didn’t work. but hyd two looks like this:
'\n\nUnfortunately, I am not able to provide more information about these potential stories as they are just brief summaries. I do not have any further details about the characters, plot, or endings of each story. These are just imaginative possibilities based on the given titles and genres. ’
hyde outputs
[print(item + '\n-------------\n') for item in hydeprompts]
Sure! “The Last Sunset” is a science fiction story set in a dystopian future where Earth is facing destruction due to a volatile atmosphere and an approaching supernova. Humanity’s last hope lies in a spacecraft called Elysium, which is headed towards a distant exoplanet called New Eden. The main character, Commander Lira, reflects on the beauty of Earth’s last sunset and wonders about the new world they are headed towards. The story explores themes of survival, sacrifice, and the unknown future.
Unfortunately, I am not able to provide more information about these potential stories as they are just brief summaries. I do not have any further details about the characters, plot, or endings of each story. These are just imaginative possibilities based on the given titles and genres.
Sure! “The Comedy of Errors” follows the hijinks and mishaps of the staff at ‘Cup O’ Laughs’ cafe as they navigate through a series of misunderstandings and mistaken identities. It’s a fast-paced comedy with a touch of romance as Tom and Julia’s relationship takes unexpected turns. The mayor’s prized Siamese cat adds to the chaos, leading to a wild adventure to avoid being caught. Humor, love, and a whole lot of confusion make this an entertaining read.
Sure! “Shadows Over Calmhaven” is a horror story set in a seemingly peaceful town called Calmhaven. However, as the nights grow longer and the fog rolls in, an ominous presence can be felt lurking in the shadows. People begin to disappear one by one, and the remaining townspeople believe that the town is cursed. Marianne, a young writer, becomes curious about the disappearances and decides to investigate. As she delves deeper, she uncovers a dark secret that has been haunting the town for years. But the more she unravels the mystery, the closer she gets to facing terrifying creatures and risking her own life. The story explores themes of supernatural horror, mystery, and the consequences of digging up buried secrets.
“Memoirs of the Heart” is a romance story set in the city of Paris. The main character, Sophia, is an artist who finds inspiration in the charming streets and cafes of the city. One day, she meets Laurent, a man with captivating eyes and a free-spirited nature. They fall deeply in love and their story mirrors the beauty of the city, with its blossoming cherry trees and vintage book stalls along the river.
However, as their love grows, they must also face the reality of their lives and the inevitability of time. As the days pass, Sophia’s heart aches as she and Laurent are slowly drawn apart. But their love remains immortalized in the memories of their Parisian spring, like a masterpiece that will never fade.
The story explores themes of love, loss, and the bittersweet nature of time. It captures the magic of Paris as a romantic backdrop for a timeless love story.
embedding-stories-hyde.ipynb
stories1 = [
"""
Title: The Last Sunset
Genre: Science Fiction
Aboard the spacecraft Elysium, Commander Lira gazed at the receding planet Earth for one last time. The atmosphere had turned volatile, and the sun was nearing supernova. Humanity's last hope lay in the stars, and Elysium's destination was a distant exoplanet named New Eden. As the final streaks of the sunset faded into the blackness of space, Commander Lira wondered what colors the sunsets on New Eden would hold.
""",
"""
Title: Through the Enchanted Forest
Genre: Fantasy
Elara, the elven mage, stepped cautiously into the whispering woods of Eldoria. Each tree hummed with ancient magic, and sprites danced in the air, leaving trails of glittering light. She had come to seek the wisdom of the Forest Oracle, a being as old as time itself. The path was winding, the challenges many, but Elara's heart was steadfast. The fate of her people rested in her hands, and she would not fail them.
""",
"""
Title: The Comedy of Errors
Genre: Comedy
It started with a misplaced coffee cup, leading to a series of absurd misunderstandings among the staff of the 'Cup O' Laughs' cafe. Tom thought Julia was in love with him, while Julia had simply mistaken his notebook for her diary. Meanwhile, the cat they thought they had adopted for the cafe was actually the mayor's prized Siamese, leading to a frantic escapade to avoid a city-wide feline search party. Hilarity ensued as love, laughter, and misplaced felines created the perfect storm of comedy.
""",
"""
Title: Shadows Over Calmhaven
Genre: Horror
Calmhaven had always been a quiet town, but when the nights grew longer and the fog rolled in, an ominous presence could be felt lurking in the shadows. The townspeople whispered of a curse, and one by one, they disappeared, leaving only silence in their wake. It was up to Marianne, a young writer with a curious mind, to uncover the dark secret that haunted their once peaceful town. But some secrets are best left buried, and as she delved deeper, Marianne found herself facing terrors beyond her wildest nightmares.
""",
"""
Title: Memoirs of the Heart
Genre: Romance
On the cobblestone streets of Paris, amidst the quaint cafes and blossoming cherry trees, Sophia found more than just inspiration for her art; she found Laurent, with eyes as deep as the Seine and a spirit that matched her own. Their love story unfolded like the pages of the books in the bouquinistes stalls along the riverbank. But love, like art, can be both timeless and fleeting. Sophia's heart ached as the days drew them apart, yet the memories of their Parisian spring would live on, an eternal masterpiece.
"""
]
stories = [*stories1]
def makepromptnr(n):
return '\n-----------------------------------\n'.join([
*stories,
f"INSTRUCTION: can you tell me more about story nr {n}?"
])
print(makepromptnr(1))
Title: The Last Sunset
Genre: Science Fiction
Aboard the spacecraft Elysium, Commander Lira gazed at the receding planet Earth for one last time. The atmosphere had turned volatile, and the sun was nearing supernova. Humanity's last hope lay in the stars, and Elysium's destination was a distant exoplanet named New Eden. As the final streaks of the sunset faded into the blackness of space, Commander Lira wondered what colors the sunsets on New Eden would hold.
-----------------------------------
Title: Through the Enchanted Forest
Genre: Fantasy
Elara, the elven mage, stepped cautiously into the whispering woods of Eldoria. Each tree hummed with ancient magic, and sprites danced in the air, leaving trails of glittering light. She had come to seek the wisdom of the Forest Oracle, a being as old as time itself. The path was winding, the challenges many, but Elara's heart was steadfast. The fate of her people rested in her hands, and she would not fail them.
-----------------------------------
Title: The Comedy of Errors
Genre: Comedy
It started with a misplaced coffee cup, leading to a series of absurd misunderstandings among the staff of the 'Cup O' Laughs' cafe. Tom thought Julia was in love with him, while Julia had simply mistaken his notebook for her diary. Meanwhile, the cat they thought they had adopted for the cafe was actually the mayor's prized Siamese, leading to a frantic escapade to avoid a city-wide feline search party. Hilarity ensued as love, laughter, and misplaced felines created the perfect storm of comedy.
-----------------------------------
Title: Shadows Over Calmhaven
Genre: Horror
Calmhaven had always been a quiet town, but when the nights grew longer and the fog rolled in, an ominous presence could be felt lurking in the shadows. The townspeople whispered of a curse, and one by one, they disappeared, leaving only silence in their wake. It was up to Marianne, a young writer with a curious mind, to uncover the dark secret that haunted their once peaceful town. But some secrets are best left buried, and as she delved deeper, Marianne found herself facing terrors beyond her wildest nightmares.
-----------------------------------
Title: Memoirs of the Heart
Genre: Romance
On the cobblestone streets of Paris, amidst the quaint cafes and blossoming cherry trees, Sophia found more than just inspiration for her art; she found Laurent, with eyes as deep as the Seine and a spirit that matched her own. Their love story unfolded like the pages of the books in the bouquinistes stalls along the riverbank. But love, like art, can be both timeless and fleeting. Sophia's heart ached as the days drew them apart, yet the memories of their Parisian spring would live on, an eternal masterpiece.
-----------------------------------
INSTRUCTION: can you tell me more about story nr 1?
# Create an array of prompts, calling 'makepromptnr' for each story
prompts = [makepromptnr(i) for i in range(1, len(stories)+1)]
from openai import OpenAI
client = OpenAI()
responses = client.completions.create(
model="gpt-3.5-turbo-instruct",
prompt=prompts,
temperature=1,
max_tokens=256,
top_p=1,
frequency_penalty=0,
presence_penalty=0
)
print(len(responses.choices))
print(responses.choices[0].text)
hydeprompts = [item.text for item in responses.choices]
hydeprompts[0]
5
Sure! "The Last Sunset" is a science fiction story set in a dystopian future where Earth is facing destruction due to a volatile atmosphere and an approaching supernova. Humanity's last hope lies in a spacecraft called Elysium, which is headed towards a distant exoplanet called New Eden. The main character, Commander Lira, reflects on the beauty of Earth's last sunset and wonders about the new world they are headed towards. The story explores themes of survival, sacrifice, and the unknown future.
'Sure! "The Last Sunset" is a science fiction story set in a dystopian future where Earth is facing destruction due to a volatile atmosphere and an approaching supernova. Humanity\'s last hope lies in a spacecraft called Elysium, which is headed towards a distant exoplanet called New Eden. The main character, Commander Lira, reflects on the beauty of Earth\'s last sunset and wonders about the new world they are headed towards. The story explores themes of survival, sacrifice, and the unknown future.'
embeddingcandidates = [*stories, *hydeprompts]
from openai import OpenAI
client = OpenAI()
output = client.embeddings.create(
model="text-embedding-ada-002",
input=embeddingcandidates,
encoding_format="float"
)
len(output.data)
storyemb, promptemb = output.data[:len(output.data)//2], output.data[len(output.data)//2:]
storyemb[0].embedding
[0.038123354,
-0.014216197,
-0.01672248,
...]
# Jupyter cell
import numpy as np
import matplotlib.pyplot as plt
from scipy.spatial.distance import cosine
# Assuming `storyemb` and `promptemb` are sequences where each item has an `.embedding` attribute containing the embeddings
# Extract embeddings and convert to numpy array
story_embeddings_array = np.array([item.embedding for item in storyemb])
prompt_embeddings_array = np.array([item.embedding for item in promptemb])
# Calculate cosine similarity matrix
similarity_matrix = np.zeros((len(story_embeddings_array), len(prompt_embeddings_array)))
for i in range(len(story_embeddings_array)):
for j in range(len(prompt_embeddings_array)):
# Use 1 - cosine distance to get similarity
similarity_matrix[i, j] = 1 - cosine(story_embeddings_array[i], prompt_embeddings_array[j])
# Set up the matplotlib figure
fig, ax = plt.subplots(figsize=(8, 6))
# Plot the heatmap
cax = ax.matshow(similarity_matrix, cmap='viridis')
# Add color bar for reference
fig.colorbar(cax)
# Annotate the heatmap with the similarity values
for i in range(len(story_embeddings_array)):
for j in range(len(prompt_embeddings_array)):
ax.text(j, i, round(similarity_matrix[i, j], 2), ha='center', va='center', color='w')
# Add titles and labels
plt.title('Cosine Similarity Heatmap Between Embeddings')
plt.xlabel('prompt_embeddings_array')
plt.ylabel('story_embeddings_array')
# Set tick labels if needed
ax.set_xticklabels([''] + list(range(similarity_matrix.shape[1])))
ax.set_yticklabels([''] + list(range(similarity_matrix.shape[0])))
plt.grid(False) # Hide the grid lines for better visibility of the heatmap cells
plt.show()
<ipython-input-22-ffc49ffe1d56>:39: UserWarning: FixedFormatter should only be used together with FixedLocator
ax.set_xticklabels([''] + list(range(similarity_matrix.shape[1])))
<ipython-input-22-ffc49ffe1d56>:40: UserWarning: FixedFormatter should only be used together with FixedLocator
ax.set_yticklabels([''] + list(range(similarity_matrix.shape[0])))
hydeprompts[1]
'\n\nUnfortunately, I am not able to provide more information about these potential stories as they are just brief summaries. I do not have any further details about the characters, plot, or endings of each story. These are just imaginative possibilities based on the given titles and genres. '
[print(item + '\n-------------\n') for item in hydeprompts]
Sure! "The Last Sunset" is a science fiction story set in a dystopian future where Earth is facing destruction due to a volatile atmosphere and an approaching supernova. Humanity's last hope lies in a spacecraft called Elysium, which is headed towards a distant exoplanet called New Eden. The main character, Commander Lira, reflects on the beauty of Earth's last sunset and wonders about the new world they are headed towards. The story explores themes of survival, sacrifice, and the unknown future.
-------------
Unfortunately, I am not able to provide more information about these potential stories as they are just brief summaries. I do not have any further details about the characters, plot, or endings of each story. These are just imaginative possibilities based on the given titles and genres.
-------------
Sure! "The Comedy of Errors" follows the hijinks and mishaps of the staff at 'Cup O' Laughs' cafe as they navigate through a series of misunderstandings and mistaken identities. It's a fast-paced comedy with a touch of romance as Tom and Julia's relationship takes unexpected turns. The mayor's prized Siamese cat adds to the chaos, leading to a wild adventure to avoid being caught. Humor, love, and a whole lot of confusion make this an entertaining read.
-------------
Sure! "Shadows Over Calmhaven" is a horror story set in a seemingly peaceful town called Calmhaven. However, as the nights grow longer and the fog rolls in, an ominous presence can be felt lurking in the shadows. People begin to disappear one by one, and the remaining townspeople believe that the town is cursed. Marianne, a young writer, becomes curious about the disappearances and decides to investigate. As she delves deeper, she uncovers a dark secret that has been haunting the town for years. But the more she unravels the mystery, the closer she gets to facing terrifying creatures and risking her own life. The story explores themes of supernatural horror, mystery, and the consequences of digging up buried secrets.
-------------
"Memoirs of the Heart" is a romance story set in the city of Paris. The main character, Sophia, is an artist who finds inspiration in the charming streets and cafes of the city. One day, she meets Laurent, a man with captivating eyes and a free-spirited nature. They fall deeply in love and their story mirrors the beauty of the city, with its blossoming cherry trees and vintage book stalls along the river.
However, as their love grows, they must also face the reality of their lives and the inevitability of time. As the days pass, Sophia's heart aches as she and Laurent are slowly drawn apart. But their love remains immortalized in the memories of their Parisian spring, like a masterpiece that will never fade.
The story explores themes of love, loss, and the bittersweet nature of time. It captures the magic of Paris as a romantic backdrop for a timeless love story.
-------------
[None, None, None, None, None]