Assistants looking at other assistants files

I´m using the assistants api and have multiple assistants each with different files.

I´m currently doing some tests on how well they are performing so after each answer I ask them for a direct quote from where they got the information and for the files they got that information from.

Somehow, in one of the rounds the assistant accessed files from another assistant.

I´m doing this through an interface I created and after checking the logs, the assistant (I can see the assistant´s id) does seem to have accessed another assistant’s files somehow.

Couldn´t find this on the forum and wanted to know if this has happened to anyone else. I haven´t been able to reproduce this yet.

Hey there and welcome to the community!

Do you think there might be some off-chance they already had access to the same files unintentionally? What does your code look like?

Some devs reported something similar previously. I want to ask if you are using the same thread or not but based on the previous posts, it seem not to matter and the bug can still occur even if different threads.

2 Likes

My code looks like this:

response= globals.Ask_question_to_thread(question,globals.thread)
response= globals.Ask_question_to_thread(question,globals.thread)
response= delete_sursa(response)
if shared.settings[“say_sources”]:
response+= "\nQuote: "+delete_sursa(globals.Ask_question_to_thread(“Tell me the exact quote/s you got that from”,globals.thread))
response+= "\nSursa: "+delete_sursa(globals.Ask_question_to_thread(“Tell me the file/s you got that from”,globals.thread))
yield response

delete_sursa is just a function to delete the [source] that´s useless right now from the api (some regex).
The relevant one may be globals.Ask_question_to_thread, which is just:

def _Ask_question_to_thread(question,thread):
curr_messages=client.beta.threads.messages.list(
thread_id = thread.id
)
length_history=len(curr_messages.data)

message = client.beta.threads.messages.create(
    thread_id=thread.id,
    role="user",
    content=question
    )

run = client.beta.threads.runs.create(
        thread_id=thread.id,
        assistant_id=current_assistant_key,
    )

messages=client.beta.threads.messages.list(
  thread_id = thread.id
)

i=0
#If we ever build a chatbot we can just check if it has increased, now we just check if it has answered the question (which is just when len>1)
while len(messages.data)==length_history+1 and i<100:
  sleep(1)
  messages=client.beta.threads.messages.list(
    thread_id = thread.id
  )
  #print(messages.data)
  i+=1
  if i%10==0:
      print("Waiting for the assistant to answer the question,i=",i)

response = messages.data[0].content[0].text.value
iter=0
#We´re doing this loop because it was giving us empty answers, and it seemed that waiting a bit gave us the answer
while len(response)==0 and iter<100:
  sleep(1)
  messages=client.beta.threads.messages.list(
    thread_id = thread.id
  )
  response = messages.data[0].content[0].text.value
  
  iter+=1
  if iter%10==0:
      print("Waiting for the assistant to answer the question,iter=",iter)
print("")
print("messages",messages)

print("the message should be:", messages.data[0].content[0].text.value)
return messages.data[0].content[0].text.value

You can see we´ve got the assistant key as a global variable, and we´ve also got a print statement of the “messages” variable. This allows us to see what has been going on (each time we use this function we see the whole history).

For simplicity I´ll only show the messages output in the cmd for the last question (files question), as it contains all 3 questions (original question, quote question and source question).

messages SyncCursorPage[ThreadMessage](data=

[ThreadMessage(id=‘msg_dSEePsI5YF4CETAt5le1JxD1’, assistant_id=‘asst_ivhh4rSbufDqiXhKAvw2s77U’, content=[MessageContentText(text=Text(annotations=,

value=‘Am găsit această informație în fișierul “196_Conditii generale AQUA_Ed a 3a_Aprilie 2019_valabile de la 01 07 2019.txt”【21†source】.’), type=‘text’)], created_at=1707315235, file_ids=, metadata={}, object=‘thread.message’, role=‘assistant’, run_id=‘run_Zmg6TQn6r3EZLIvKLInk847a’, thread_id=‘thread_KkuQHUPDxr0lKzWtthrCsuJ0’),

ThreadMessage(id=‘msg_sCduqOVxHB4NG7BB7SmfNH6f’, assistant_id=None, content=[MessageContentText(text=Text(annotations=, value=‘Spune-mi fișierele din care ai luat-o’), type=‘text’)], created_at=1707315234, file_ids=, metadata={}, object=‘thread.message’, role=‘user’, run_id=None, thread_id=‘thread_KkuQHUPDxr0lKzWtthrCsuJ0’),

ThreadMessage(id=‘msg_XOrrazlaqjMXAiVOve4rawsR’, assistant_id=‘asst_ivhh4rSbufDqiXhKAvw2s77U’, content=[MessageContentText(text=Text(annotations=,

value=‘This was the exact quote from the document (which it shouldn´t have had access to)’), type=‘text’)], created_at=1707315230, file_ids=, metadata={}, object=‘thread.message’, role=‘assistant’, run_id=‘run_Gn7mHQnqFI842yfgHFDShxS9’, thread_id=‘thread_KkuQHUPDxr0lKzWtthrCsuJ0’),

ThreadMessage(id=‘msg_9Nk00pbNtidIyYD65afYPOy4’, assistant_id=None, content=[MessageContentText(text=Text(annotations=,

value=‘Spune-mi citatul exact din care ai luat-o’), type=‘text’)], created_at=1707315229, file_ids=, metadata={}, object=‘thread.message’, role=‘user’, run_id=None, thread_id=‘thread_KkuQHUPDxr0lKzWtthrCsuJ0’),

ThreadMessage(id=‘msg_5yBDuBumj9jSlQRyjf0YjD2a’, assistant_id=‘asst_ivhh4rSbufDqiXhKAvw2s77U’, content=[MessageContentText(text=Text(annotations=,

value=‘Here was the answer of the assistant (which is based on the document it talks about later (not useful for our question))’), type=‘text’)], created_at=1707315226, file_ids=, metadata={}, object=‘thread.message’, role=‘assistant’, run_id=‘run_XnNJwz37W0t1cTDes5bJ1bP3’, thread_id=‘thread_KkuQHUPDxr0lKzWtthrCsuJ0’),

ThreadMessage(id=‘msg_mHZXwS3iyFDV7XrFDeYgtEDP’, assistant_id=None, content=[MessageContentText(text=Text(annotations=,

value=‘Care sunt situatiile in care poate inceta contractul de asigurare?’), type=‘text’)], created_at=1707315221, file_ids=, metadata={}, object=‘thread.message’, role=‘user’, run_id=None, thread_id=‘thread_KkuQHUPDxr0lKzWtthrCsuJ0’)], object=‘list’, first_id=‘msg_dSEePsI5YF4CETAt5le1JxD1’, last_id=‘msg_mHZXwS3iyFDV7XrFDeYgtEDP’, has_more=False)

the message should be: Am găsit această informație în fișierul “196_Conditii generale AQUA_Ed a 3a_Aprilie 2019_valabile de la 01 07 2019.txt”【21†source】.

The thing to look at here basically is that they have the same assistant_id and same thread_id (also annotations is still broken -.-).

There is an additional possibility. I think I created this assistant as a copy of the AQUA assistant and then deleted the files. It´s possible that the bug may be in the deletion of the files.

They used the same thread for all 3 questions.