Issue with Functions and File Retrieval in API Integration

API
, ,
1

Hello OpenAI developer community,

I’m facing a technical challenge integrating Assistant API functionalities into my product and would appreciate your assistance or suggestions.

The specific problem I am encountering is as follows: whenever I activate the “functions” in Assistant API, the “retrieval” feature (data retrieval from files) stops working. That is, with the functions activated, I lose access to crucial data stored in files, which is essential to provide the necessary company context in the conversation.

This creates a dilemma for the launch of my product, as without access to the files, I cannot implement the desired business context. On the other hand, without the functions, I lose the ability to retrieve important information during interactions.

I have conducted tests in the OpenAI playground and noticed that this behavior repeats itself, regardless of whether I am using the GPT-4.1.1-06 or GPT-3.5 model.

I wonder if anyone has experienced similar behavior or knows how to resolve this issue. Any guidance or suggestion would be very welcome, as this is directly affecting the launch schedule of my product.

Thank you in advance for any help you can offer.

Sincerely,
Marco Araujo

1 Like
2

Seems to be working for me in the playground.

Screenshot 2023-12-01 at 01.49.44
Screenshot 2023-12-01 at 01.49.44950×1024 37.4 KB

3

I can configure everything normally, but what’s not working is the access to the file data. For instance, I placed a contact phone number inside a file. When I activate the functions and open a new thread, it doesn’t find the number. However, when I deactivate the functions and try again in another thread, it works normally.

1 Like
4

“When I activate the functions and open a new thread, it doesn’t find the number.”

I haven’t tried this configuration out, but I wonder why you are opening a new thread. If you use one thread for file retrieval and another for function calls then I would expect the behaviour you describe.

5

Hope you have considered these:

  1. function call is interfering with file id or messing thread

  2. If retrieval works on one thread, it should work on all as long as files are globally scoped and not scoped to thread, that is possible via uploading to an active thread.

  1. function calling requires correct tool outputs to be submitted. Not applicable to you but do look into it.

One of these could be it. Best of luck !

6

I’m trying in the same thread and the problem persists. When possible, please test this scenario: create an assistant, upload a simple file containing contact details, for example, and enable any function.

In this scenario, within the same thread, the retrieval doesn’t work for me, only the functions do. When the functions are removed and I open a new thread, the retrieval works.

7

I have indeed considered that. What’s most interesting is that the problem presents itself even within the playground for me.

I even tried adding a random function, like the example one, “getweather”, and the behavior is the same. Once the function is activated, the retrieval stops working correctly.

8

Hi, I have the same issue.
Yesterday, it was working properly. Today, the same bot is calling my function many times without any reason… I’m doing the same test.
I’m wondering if something changed in OpenAI.

1 Like
9

OpenAI’s service terms were updated November 6, the day assistants was released - likely informed by what they knew about the quality of “assistants”

2. Beta Services

This section governs your use of services or features that OpenAI offers on an alpha, preview, early access, or beta basis (“Beta Services”). Beta Services are offered “as-is” to allow testing and evaluation and are excluded from any indemnification obligations OpenAI may have to you.

OpenAI makes no representations or warranties for Beta Services, including any warranty that Beta Services will be generally available, uninterrupted or error-free, or that Content will be secure or not lost or damaged. Except to the extent prohibited by law, OpenAI expressly disclaims all warranties for Beta Services, including any implied warranties of merchantability, satisfactory quality, fitness for a particular purpose, non-infringement, or quiet enjoyment, and any warranties arising out of any course of dealing or usage of trade.

10

I can confirm that the retrieval and function call are working. I tried a very simple example.
(1) Text file with a single contact address
(2) Three function calls get_lat, get_long, get_display name (get latitude, longitude and display name).

I am having some difficulties with my code, but it’s on my end. Originally I had written this code to retrieve a single address from a prompt but now I am asking it to retrieve it from a contacts file. It changes the behaviour of the function calling. When I was getting the address from a prompt, I would get a ‘require action’ status one time for all three function calls. For some reason, when I get the same information for a contact file, I get a ‘require action’ status but it only specifies 1 function call get_lat, then I run it again passing in the results and it goes into ‘requires action’ status again but this time it wants me to run get_long.

That said, I can see that it is correctly parsing the address information from the contacts file and passing it back to be used as an argument for the function calling. I just to refactor my code so that if I get more than one ‘require action’ status I will repeat until all function calls are made.

11

This might just be because we are working with Beta code or am I might be missing something. Combining function calling with file retrieval does seem to be having a strange behaviour.

Without File Retrieval
I wrote some code a while back that defines three functions to get the latitude, longitude and a display name for a given address. I provide the address in a prompt. I provide the prompt and it consistently puts the run object into a ‘requires action’ status. The section of the run object looks like this:

'required_action': {'submit_tool_outputs': {'tool_calls': [{'id': 'call_FLf8qdNeIh7YC3tauBfTkQ1d',
     'function': {'arguments': '{"location": "Cobourg, Ontario"}',
      'name': 'get_lat'},
     'type': 'function'},
    {'id': 'call_3HrT4gh7oX8Yh2gMirC6Lq2y',
     'function': {'arguments': '{"location": "Cobourg, Ontario"}',
      'name': 'get_long'},
     'type': 'function'},
    {'id': 'call_Z6lrL2GDeCoXzBKkQ23IFodR',
     'function': {'arguments': '{"location": "Cobourg, Ontario"}',
      'name': 'get_display_name'},
     'type': 'function'}]},
  'type': 'submit_tool_outputs'},
 'started_at': 1701577250,
 'status': 'requires_action',

You can see the three function names ‘get_lat’, ‘get_long’, and get_display_name and the argument to pass to them that was pulled from the prompt of my full address.

With File Retrieval

In the next iteration, I used file retrieval to pull the location from a plain text file that had a single contact name and address in it. Here is what the same section of the run object looked like. It’s quite muddled and I can’t find this structure documented anywhere. I suspect its simply a matter that we are dealing with beta code and OpenAI may be in the process of changing the structure of the run object.

'required_action': {'submit_tool_outputs': {'tool_calls': [{'id': 'call_O5hqVnwak0UWOEGqWRUQEKt7',
     'function': {'arguments': '{\n  "tool_uses": [\n    {\n      "recipient_name": "functions.get_lat",
 \n      "parameters": {\n        "location": "999 William St, Cobourg, Ontario"\n      }\n    },\n    
{\n      "recipient_name": "functions.get_long",
\n      "parameters": {\n        "location": "999 William St, Cobourg, Ontario"\n      }\n    }\n  ]\n}',
      'name': 'parallel'},
     'type': 'function'}]},
  'type': 'submit_tool_outputs'},
 'started_at': 1701548209,
 'status': 'requires_action',

I have not defined a function with the name of parallel and I can see my function names of get_lat and get_long in the strings “functions.get_lat” and “functions.get_long”. Interesting, albeit a little wonky right now.

EDIT: Over time, I continued to get this kind of undocumented run object, not every time but frequently enough to be annoying. In the end, I simply added the instruction “Do not do function calling in parallel”. I can’t say for sure, but it seems to work.

12

From my experience so far, if functions are used together with retrieval, the assistant does not know how to read the content from the uploaded file.
For example, following scenario:

User: Hello, how you can help?
Assistant: I see you’ve uploaded a file. How can I assist you with it?
User: What is in the file?
Assistant: It seems that I’m unable to directly access the contents of the file you’ve uploaded using the available tool. However, if you have any specific questions or if there’s anything you’d like me to do with the file, feel free to let me know and I’ll do my best to assist you based on the information you provide.

In the above query, assistant tried to call “open_url” function, which is not specified in the functions list I have set for assistant.
“function_calls”: [
{
“function_name”: “open_url”,
“arguments”: “{"url":"file-XXXX"}”,
“response”: “{"tool_call.function_error": "open_url", "error": "Function is not available."}”
}]
However, seems that functions I have defined work, but cannot use the knowledge that is configured in assistant creation along with the data functions provide. Also if I only use retrieval as tool, then assistant is able to read the content.

2 Likes
13

I wonder if this issue is being investigated on OpenAI side? I can reproduce this consistently, so if there is any help needed to provide traces or other debugging information, I would gladly do so.

EDIT: Seems like, with latest GPT4 model retrieval + function tools work together better and assistant is able to retrieve data from knowledge and use that information together with data retrieved from function calls. Will be testing more now.

1 Like
14

I don’t know if they are investigating it, do you know if there is a way to contact them?

15

I have understood this dev forum is the official way to report issues to OpenAI, even though I have not seen here clearly official OpenAI responses yet.

Under OpenAI Python SDK GitHub repository issues creation template there is following.

“Confirm this is an issue with the Python library and not an underlying OpenAI API
Issues with the underlying OpenAI API should be reported on our Developer Community”, which is this place.

16

I hear you’re dealing with a tricky situation integrating Assistant API functionalities. That’s a head-scratcher for sure. I haven’t faced the exact issue.

17

I noticed that when I enabled functions, the Assistant started asking me for files (one of which I do intend to provide “dynamically”). I did expect it to still utilize the files I uploaded via the UI.

I’m thinking - once you enable functions, the files uploaded via the UI might not be in play.

Maybe try creating files manually once functions are enabled: https://platform.openai.com/docs/api-reference/files/create

Also, it would be good to know the output from list files:
https://platform.openai.com/docs/api-reference/files/list

And I guess the runs object has a files array that shows files used in the run - probably be good to keep an eye on that.

I need this working myself so I may be facing the same issue soon.

18

Any updates with that? I am having the same problem

19

My original problem when using file retrieval and function calling was that sometimes, but not always the required action returned to me would not be the names of the function to be called but it would be ‘parallel’. This is an undocumented response.

I tried adding the following as part of my instructions.

‘Do not run function calls in parallel’

. Example is below. I have not done extensive testing

 Add a message to the thread 
message = client.beta.threads.messages.create(
    thread_id=thread.id,
    role="user",
    content="Get one or more addresses from the file and determine latitude and longitude and name to display. Do not run function calls in parallel"
)

Since then, the code works as expected. The file is read, the address is read from the file and passed as parameters to functions that determine latitude, longitude and a display name. It’s a very simple example, but it demonstrates file retrieval and function calling working together.

20

I updated my assistant to the gpt3.5-turbo-0125 model, I put this same instruction and it partially worked, file recovery is working but the functions are a little messed up, for example: I inserted a function to capture address data for product delivery , and instead of asking the user this, the assistant is automatically filling in random data. I believe that the central problem of this issue has been resolved, but the use of functions together with the assistants is not yet functional.

1 Like