Streaming markdown text and images from assistant using code interpreter

suspicious_cow · June 16, 2024, 6:15pm

So, I’ve been banging my head up against this one for a while now. I’ve got code that will format markdown in a vscode jupyter notebook cell as it is streaming and now I’m trying to get the images from code interpreter as part of the output. I want the images to be inline with the text just like we see in ChatGPT. Nothing I’m trying so far works. Any thoughts?

Here is my current code that handles markdown:

class EventHandler(AssistantEventHandler):
    """Custom event handler for processing assistant events."""

    def __init__(self):
        super().__init__()
        self.results = []  # Initialize the results list

    @override
    def on_text_delta(self, delta, snapshot):
        """Handle the event when there is a text delta (partial text)."""
        self.results.append(delta.value)
        self.update_output()

    def update_output(self):
        """Update the Jupyter Notebook cell with the current markdown content."""
        clear_output(wait=True)
        markdown_content = "".join(self.results)
        display(Markdown(markdown_content))


assistant_thread = client.beta.threads.create(
    messages=[
        {
            "role": "user",
            "content": "Give me a summary of the file penguins_size.csv. With at least one small table of data and one visualization."
        },
    ]
)

with client.beta.threads.runs.stream(
    thread_id=assistant_thread.id,
    assistant_id=assistant.id,
    instructions="""
    You are a helpful assistant.
    """,
    event_handler=EventHandler(),
) as stream:
    stream.until_done()

PaulBellow · June 16, 2024, 6:28pm

Is it giving you errors or just not working?

I believe you might need to download them to your system then insert them? I’ve not worked a lot with assistants, though.

Watching with interest, though. Hopefully someone smarter comes along.

suspicious_cow · June 17, 2024, 10:17am

Yeah that’s the issue. It is definitely producing the files as images while the stream is happening (have verified this by looking at storage in the API UI) and I can absolutely show the files after the stream is done but the tricky part appears to be showing the files while the stream is happening so they appear in the context that produced them.

suspicious_cow · June 22, 2024, 12:00am

Hopefully someone can take the ball and run with it. This is as far as I was able to get. This code will stream and show text and make an image but will not display the image.

class EventHandler(AssistantEventHandler):
    """Custom event handler for processing assistant events."""

    def __init__(self):
        super().__init__()
        self.results = []  # Initialize an empty list to store the results
        self.image_counter = 0  # Counter for images

    @override
    def on_text_delta(self, delta, snapshot):
        """Handle the event when there is a text delta (partial text)."""
        # Append the delta value (partial text) to the results list
        self.results.append(delta.value)
        # Call the method to update the Jupyter Notebook cell
        self.update_output()

    @override
    def on_image_file_created(self, image_file):
        """Handle the event when an image file is created."""
        print(f"Image file created: {image_file.file_id}")
        # Retrieve the image file information and content
        file_info = client.files.retrieve(image_file.file_id)
        image_content = client.files.content(file_info.id).content
        image_base64 = base64.b64encode(image_content).decode('utf-8')
        # Append the image HTML to the results list
        self.results.append(f'![Visualization {self.image_counter}](data:image/png;base64,{image_base64})\n')
        self.image_counter += 1
        # Call the method to update the Jupyter Notebook cell
        self.update_output()

    def update_output(self):
        """Update the Jupyter Notebook cell with the current markdown content."""
        # Clear the current output in the Jupyter Notebook cell
        clear_output(wait=True)
        # Join all the text and image fragments stored in results to form the complete Markdown content
        markdown_content = ''.join(self.results)
        # Display the Markdown content in the Jupyter Notebook cell
        display(Markdown(markdown_content))

# Create a thread to send a message and get output
assistant_thread = client.beta.threads.create(
    messages=[
        {
            "role": "user",
            "content": (
                "Give me a one paragraph summary of the file penguins_size.csv. "
                "With at least one small table of data and one visualization."
            )
        },
    ]
)

# Stream the output from the assistant
with client.beta.threads.runs.stream(
    thread_id=assistant_thread.id,  # Use the ID of the created thread
    assistant_id=assistant.id,  # Use the ID of the assistant
    event_handler=EventHandler(),  # Use the custom event handler to process events
) as stream:
    stream.until_done()  # Continue streaming until the assistant has finished responding

suspicious_cow · June 22, 2024, 12:16pm

Okay, now I’m really confused. At first, I thought it was me but I went back to the original streaming code to start over and discovered that the documentation is flawed. The key to success appears to be in the event handler. The documentation shows this code sample for streaming:

from typing_extensions import override
from openai import AssistantEventHandler, OpenAI
 
client = OpenAI()
 
class EventHandler(AssistantEventHandler):
    @override
    def on_text_created(self, text) -> None:
        print(f"\nassistant > ", end="", flush=True)

    @override
    def on_tool_call_created(self, tool_call):
        print(f"\nassistant > {tool_call.type}\n", flush=True)

    @override
    def on_message_done(self, message) -> None:
        # print a citation to the file searched
        message_content = message.content[0].text
        annotations = message_content.annotations
        citations = []
        for index, annotation in enumerate(annotations):
            message_content.value = message_content.value.replace(
                annotation.text, f"[{index}]"
            )
            if file_citation := getattr(annotation, "file_citation", None):
                cited_file = client.files.retrieve(file_citation.file_id)
                citations.append(f"[{index}] {cited_file.filename}")

        print(message_content.value)
        print("\n".join(citations))


# Then, we use the stream SDK helper
# with the EventHandler class to create the Run
# and stream the response.

with client.beta.threads.runs.stream(
    thread_id=thread.id,
    assistant_id=assistant.id,
    instructions="Please address the user as Jane Doe. The user has a premium account.",
    event_handler=EventHandler(),
) as stream:
    stream.until_done()

The problem, however, is that this code doesn’t actually stream. I know the documentation has a reputation of being bad but I had no idea it would rise to this level.

After much playing around I arrived at this solution for the event handler to actually stream the output:

class EventHandler(AssistantEventHandler):
    """Custom event handler for processing assistant events."""

    def __init__(self):
        super().__init__()
        self.results = []  # Initialize the results list

    @override
    def on_text_created(self, text) -> None:
        """Handle the event when text is first created."""
        # Print the created text to the console
        print("\nassistant text > ", end="", flush=True)
        # Append the created text to the results list
        self.results.append(text)

    @override
    def on_text_delta(self, delta, snapshot):
        """Handle the event when there is a text delta (partial text)."""
        # Print the delta value (partial text) to the console
        print(delta.value, end="", flush=True)
        # Append the delta value to the results list
        self.results.append(delta.value)

    def on_tool_call_created(self, tool_call):
        """Handle the event when a tool call is created."""
        # Print the type of the tool call to the console
        print(f"\nassistant tool > {tool_call.type}\n", flush=True)

    def on_tool_call_delta(self, delta, snapshot):
        """Handle the event when there is a delta (update) in a tool call."""
        if delta.type == 'code_interpreter':
            # Check if there is an input in the code interpreter delta
            if delta.code_interpreter.input:
                # Print the input to the console
                print(delta.code_interpreter.input, end="", flush=True)
                # Append the input to the results list
                self.results.append(delta.code_interpreter.input)
            # Check if there are outputs in the code interpreter delta
            if delta.code_interpreter.outputs:
                # Print a label for outputs to the console
                print("\n\noutput >", flush=True)
                # Iterate over each output and handle logs specifically
                for output in delta.code_interpreter.outputs or []:
                    if output.type == "logs":
                        # Print the logs to the console
                        print(f"\n{output.logs}", flush=True)
                        # Append the logs to the results list
                        self.results.append(output.logs)

# Using our first assistant
with client.beta.threads.runs.stream(
    thread_id=thread.id,
    assistant_id=assistant.id,
    event_handler=EventHandler(),
) as stream:
    stream.until_done()

Which gives me this result:

assistant tool > file_search

assistant text > The main characters in “The Wonderful Wizard of Oz” are:

Dorothy - A young girl from Kansas who is transported to the Land of Oz by a cyclone【10:6†source】.
Toto - Dorothy’s small dog who accompanies her on her journey【10:8†source】.
Scarecrow - A character Dorothy meets who desires to have brains【10:5†source】.
Tin Woodman - Another companion of Dorothy who wishes for a heart【10:5†source】.
Cowardly Lion - A lion who joins Dorothy in hopes of gaining courage【10:5†source】.
The Wizard of Oz - The ruler of the Emerald City who the characters believe can grant their wishes【10:5†source】.

Additionally, there are other notable characters:

Glinda, the Good Witch of the South - Who helps Dorothy and her friends【10:16†source】.
The Wicked Witch of the West - One of the main antagonists in the story【10:12†source】【10:13†source】.

Now the adventure begins. I will keep digging and find out how to deal with the annotations properly and post back here. My theory is that this will lead to the answer for the original question as, apparently, code interpreter file information is given as an annotation. We will see. Stay tuned

PaulBellow · June 22, 2024, 12:17pm

Nice. Glad you got it sorted.

Feel free to post in Documentation about the error, and we’ll try to pass it along or get it seen. Not sure if there’s a better way to report it…

suspicious_cow · June 22, 2024, 12:18pm

Thanks @PaulBellow will do

suspicious_cow · June 22, 2024, 2:55pm

Continuing on my journey. I had the Wizard of Oz as one big text file so decided to use something else with slightly more structure. I opted for a pdf version of Dracula. Here is the full updated code:

# Import necessary libraries
from openai import OpenAI  # Used for interacting with OpenAI's API
from typing_extensions import override  # Used for overriding methods in subclasses
from openai import AssistantEventHandler  # Used for handling events related to OpenAI assistants

# Create an instance of the OpenAI class to interact with the API.
# This assumes you have set the OPENAI_API_KEY environment variable.
client = OpenAI() 

class EventHandler(AssistantEventHandler):
    """Custom event handler for processing assistant events."""

    def __init__(self):
        super().__init__()
        self.results = []  # Initialize the results list

    @override
    def on_text_created(self, text) -> None:
        """Handle the event when text is first created."""
        # Print the created text to the console
        print("\nassistant text > ", end="", flush=True)
        # Append the created text to the results list
        self.results.append(text)

    @override
    def on_text_delta(self, delta, snapshot):
        """Handle the event when there is a text delta (partial text)."""
        # Print the delta value (partial text) to the console
        print(delta.value, end="", flush=True)
        # Append the delta value to the results list
        self.results.append(delta.value)

    def on_tool_call_created(self, tool_call):
        """Handle the event when a tool call is created."""
        # Print the type of the tool call to the console
        print(f"\nassistant tool > {tool_call.type}\n", flush=True)

    def on_tool_call_delta(self, delta, snapshot):
        """Handle the event when there is a delta (update) in a tool call."""
        if delta.type == 'code_interpreter':
            # Check if there is an input in the code interpreter delta
            if delta.code_interpreter.input:
                # Print the input to the console
                print(delta.code_interpreter.input, end="", flush=True)
                # Append the input to the results list
                self.results.append(delta.code_interpreter.input)
            # Check if there are outputs in the code interpreter delta
            if delta.code_interpreter.outputs:
                # Print a label for outputs to the console
                print("\n\noutput >", flush=True)
                # Iterate over each output and handle logs specifically
                for output in delta.code_interpreter.outputs or []:
                    if output.type == "logs":
                        # Print the logs to the console
                        print(f"\n{output.logs}", flush=True)
                        # Append the logs to the results list
                        self.results.append(output.logs)

drac_file = client.files.create(file=open("./artifacts/Dracula.pdf","rb"), purpose="assistants")

# Create an assistant using the client library.
try:
    assistant = client.beta.assistants.create(
        model="gpt-4o",  # Specify the model to be used.
        instructions=(
            "You are a helpful assistant that answers questions about the stories in your files. "
            "The stories are from a variety of authors. "
            "You will answer questions from the user about the stories. All you will do is answer questions about the stories in the files and provide related information. "
            "If the user asks you a question that is not related to the stories in the files, you should let them know that you can only answer questions about the stories."
        ),
        name="Quick Assistant and Vector Store at Once",  # Give the assistant a name.
        tools=[{"type": "file_search"}],  # Add the file search capability to the assistant.
        # Create a vector store and attach it to the assistant in one step.
        tool_resources={
            "file_search": {
                "vector_stores": [
                    {
                        "name": "Vector Store Auto Attached to Assistant",
                        "file_ids": [
                            drac_file.id,
                        ],
                        "metadata": {
                            "Book1": "Wizard of Oz", 
                        }
                    }
                ]
            }
        },
        metadata={  # Add metadata about the assistant's capabilities.
            "can_be_used_for_file_search": "True",
            "has_vector_store": "True",
        },
        temperature=1,  # Set the temperature for response variability.
        top_p=1,  # Set the top_p for nucleus sampling.
    )
except Exception as e:
    print(f"An error occurred while creating the assistant: {e}")
else:
    # Print the details of the created assistant to check its properties.
    print(assistant)  # Print the full assistant object.
    print("\n\n")
    print("Assistant Name: " + assistant.name)  # Print the name of the assistant.
    print("\n")
    
    # get the vector store information
    unnamed_assistant_vector_store = client.beta.vector_stores.retrieve(assistant.tool_resources.file_search.vector_store_ids[0])
    print("Vector Store Name: " + str(unnamed_assistant_vector_store.name))
    print("Vector Store Id: " + unnamed_assistant_vector_store.id)
    print("Vector Store Metadata: " + str(unnamed_assistant_vector_store.metadata))

# Always name your vector stores
updated_vector_store = client.beta.vector_stores.update(
    vector_store_id=unnamed_assistant_vector_store.id,
    name="Dracula Vector Store",
    metadata={"Book1": "Dracula"}
)

print("Vector Store Name: " + str(updated_vector_store.name))
print("Vector Store Id: " + updated_vector_store.id)
print("Vector Store Metadata: " + str(updated_vector_store.metadata))

# Create a thread and attach the file to the message
thread = client.beta.threads.create(
    messages=[
    {
    "role": "user",
    "content": "Who are all the main characters in Dracula? Cite the location they are first introduced in the book. Every character should have a separate citation.",
    }
]
)

# Using our first assistant
with client.beta.threads.runs.stream(
    thread_id=thread.id,
    assistant_id=assistant.id,
    event_handler=EventHandler(),
) as stream:
    stream.until_done()

The above code will produce a result like this:

assistant tool > file_search

assistant text > Here are the main characters in “Dracula” and the locations where they are first introduced in the book:

Jonathan Harker: Introduced in the first entry of his journal.
- “3 May. Bistritz. - Left Munich at 8:35 P.M., on 1st May…”
- 【4:3†Dracula.pdf】
Count Dracula: Introduced when Jonathan Harker meets him at his castle.
- “Within, stood a tall old man, clean shaven save for a long white moustache, and clad in black from head to foot, without a single speck of color about him anywhere.”
- 【4:11†Dracula.pdf】
Mina Murray (later Mina Harker): First mentioned in Jonathan Harker’s journal.
- “Whilst I was waiting I heard a lot of words which I could not understand, but amongst them were frequently repeated words, “Ordog” - Satan, “pokol” - hell, “stregoica” - witch, “vrolok” and “vlkoslak”…”
- 【4:8†Dracula.pdf】
Lucy Westenra: Introduced in Mina’s letter to Lucy.
- “My dearest Lucy, - Forgive my long delay in writing, but I have been simply overwhelmed with work. The life of an assistant schoolmistress is sometimes trying.”
- 【4:8†Dracula.pdf】
Dr. John Seward: Introduced through his own diary entries.
- “3 May. Bistritz. - Left Munich at 8:35 P.M., on 1st May…"
- 【4:11†Dracula.pdf】
Professor Abraham Van Helsing: Introduced in Dr. Seward’s diary when he writes to him for help regarding Lucy’s condition.
- “25 August. - A long talk with Van Helsing.”
- 【4:6†Dracula.pdf】
Arthur Holmwood (Lord Godalming): Introduced in correspondence discussing Lucy’s suitors.
- “My dearest Lucy, - Forgive my long delay in writing, but I have been simply overwhelmed with work.”
- 【4:8†Dracula.pdf】
Quincey Morris: First introduced indirectly as a suitor of Lucy, mentioned in correspondence.
- “My dearest Lucy, - Forgive my long delay in writing, but I have been simply overwhelmed with work.”
- 【4:8†Dracula.pdf】

Please let me know if you have more questions or need further details on any of these characters.

My investigation continues…

suspicious_cow · June 23, 2024, 1:01pm

Okay, I think I’ve gone as far as the API will allow me to go for now. Here is the final code for anyone that wants to play around with this idea. I managed to get some interesting information from the annotations but they just gave me character positions that were off. I am pretty sure it has to do with the way the annotations are denoted.

Here is where I ended up. I got down to the message level with this code:

message = client.beta.threads.messages.retrieve(
    thread_id=thread.id,
    message_id=client.beta.threads.messages.list(thread_id=thread.id,order="desc").data[0].id
)

print(message.content[0].text)

Which gave me this result:
Text(annotations=[FileCitationAnnotation(end_index=256, file_citation=FileCitation(file_id=‘file-ZpcgUzqAk0M6530CLcLkMT52’), start_index=244, text=‘【8:3†source】’, type=‘file_citation’), FileCitationAnnotation(end_index=375, file_citation=FileCitation(file_id=‘file-ZpcgUzqAk0M6530CLcLkMT52’), start_index=363, text=‘【8:9†source】’, type=‘file_citation’), FileCitationAnnotation(end_index=485, file_citation=FileCitation(file_id=‘file-ZpcgUzqAk0M6530CLcLkMT52’), start_index=473, text=‘【8:9†source】’, type=‘file_citation’), FileCitationAnnotation(end_index=616, file_citation=FileCitation(file_id=‘file-ZpcgUzqAk0M6530CLcLkMT52’), start_index=603, text=‘【8:12†source】’, type=‘file_citation’), FileCitationAnnotation(end_index=769, file_citation=FileCitation(file_id=‘file-ZpcgUzqAk0M6530CLcLkMT52’), start_index=756, text=‘【8:10†source】’, type=‘file_citation’), FileCitationAnnotation(end_index=914, file_citation=FileCitation(file_id=‘file-ZpcgUzqAk0M6530CLcLkMT52’), start_index=901, text=‘【8:13†source】’, type=‘file_citation’), FileCitationAnnotation(end_index=1022, file_citation=FileCitation(file_id=‘file-ZpcgUzqAk0M6530CLcLkMT52’), start_index=1009, text=‘【8:19†source】’, type=‘file_citation’), FileCitationAnnotation(end_index=1135, file_citation=FileCitation(file_id=‘file-ZpcgUzqAk0M6530CLcLkMT52’), start_index=1123, text=‘【8:0†source】’, type=‘file_citation’)], value=‘Here are the main characters in “Dracula” along with the citations for their first introduction in the text:\n\n1. Jonathan Harker:\n - Jonathan Harker is introduced through his journal entry describing his approach to Count Dracula's castle【8:3†source】.\n\n2. Mina Murray (Harker):\n - Mina Murray is introduced through a letter to her friend Lucy Westenra【8:9†source】.\n\n3. Lucy Westenra:\n - Lucy Westenra is introduced through the same letter from Mina Murray【8:9†source】.\n\n4. Count Dracula:\n - Count Dracula is introduced when Jonathan Harker meets him at his castle in Transylvania【8:12†source】.\n\n5. Dr. John Seward:\n - Dr. John Seward is introduced through his diary entries discussing Renfield and the activities at his asylum【8:10†source】.\n\n6. Arthur Holmwood (Lord Godalming):\n - Arthur Holmwood is introduced in the context of Lucy’s suitors and their engagement【8:13†source】.\n\n7. Quincey Morris:\n - Quincey Morris is introduced alongside the other suitors of Lucy【8:19†source】.\n\n8. Renfield:\n - Renfield is introduced through Dr. Seward’s observational notes in his diary【8:0†source】.\n\nThese citations provide the exact sections in “Dracula” where the main characters are first introduced.’)

I then broke down the annotations with this code:

# Extract the message content and annotations
message_text_object = message.content[0]
message_text_content = message_text_object.text.value  # Access the value attribute for the actual text
annotations = message_text_object.text.annotations  # Access annotations directly

# Print the annotations in a cleaner format
for index, annotation in enumerate(annotations):
    print(f"Annotation {index + 1}:")
    print(f"  End Index: {annotation.end_index}")
    print(f"  Start Index: {annotation.start_index}")
    print(f"  Text: {annotation.text}")
    print(f"  Type: {annotation.type}")
    if hasattr(annotation, 'file_citation'):
        file_citation = annotation.file_citation
        print(f"  File Citation:")
        print(f"    File ID: {file_citation.file_id}")
    print("")  # Add a blank line for readability

Which gave me results like this:
Annotation 1:
End Index: 256
Start Index: 244
Text: 【8:3†source】
Type: file_citation
File Citation:
File ID: file-ZpcgUzqAk0M6530CLcLkMT52

Annotation 2:
End Index: 375
Start Index: 363
Text: 【8:9†source】
Type: file_citation
File Citation:
File ID: file-ZpcgUzqAk0M6530CLcLkMT52

Annotation 3:
End Index: 485
Start Index: 473
Text: 【8:9†source】
Type: file_citation
File Citation:
File ID: file-ZpcgUzqAk0M6530CLcLkMT52

So the good news is there a a lot more information in the annotations than previously thought. The bad news is it isn’t accurate. If I actually go to the character indexes listed they don’t point to the proper references. Or, I should say, they don’t “appear” to point there. I suspect they do point to the right place if the citation that reference specific sections of the text can be deciphered properly.

For example, for Annotation one if we just go with characters 244 to 256 it doesn’t track. (Nor do words or tokens, I tried all three).

I think the secret lies in the position in the citation: 【8:3†source】

But I have been unable to decipher it…

karl.schelhammer1 · July 25, 2024, 4:58pm

@suspicious_cow, might you have any insights on how to extend the event handler to be able to pass the tokens along through a websocket or SSE connection? I want to consume the stream in my backend, and forward the tokens via the on_text_delta(…) method using the event handler.

Based on your experience, does that seem advisable or even possible?

For reference, here’s a sample of what I’ve tried:

suspicious_cow · July 28, 2024, 2:15am

Hey @karl.schelhammer1

Apologies for the late reply.

I haven’t gone down that path myself as I haven’t had a client need it. I wish I had better guidance for you.

Topic		Replies	Views
How to stream with the Assistant API and a Flask endpoint? API python , assistants-api	5	357	July 26, 2024
How can show a streaming output in a frontend and do it progressively? API chatgpt , api , assistants-api	1	505	July 2, 2024
Flask Streaming Examples? API assistants-api	2	83	September 7, 2024
Assistant API code_interpreter output not populated API assistants-api , assistants-streaming	3	369	September 18, 2024
Code interpreter with streaming enabled ends the runs too early Bugs	4	314	September 8, 2024

Streaming markdown text and images from assistant using code interpreter

Related Topics