Only MOCK functions running?

API
1

Over the past few days we noticed that the Assistants API stopped calling real functions over the API. When we “test” the assistants in the API, it is performing a “simulated function” and generating the results. It appears to be doing this over the API as well (just making up responses, effectively, but not callling tools at all).

Anyone else experiencing this?

It’s a strang edevelopment. after the model updates to make them less “lazy” there was a big improvement. And in the past day or so it basically completely stopped calling functions.

Update 1: I checked back our server logs. It was working fine until yesterday. And all of a sudden today, with no changes, it refuses to call any functions. It only hallucinates the response and provides it. And it often says there has been a “slight hiccup” in running the function, except when we see our logs it is not actually calling a tool/function at all something got broken bad.

Update 2: We have not tested with GPT4; GPT-3.5-Turbo (the new ones); and both the November and January GPT4-Turbo models. So ALL models have stopped calling functions/tools even when explicitly instructed to do so. And in the playground are using code intepreter with markdown code to “hallucinate” a function call.

Update 3: When trying to duplicate Assistants to test, you now get an error "unknown parameter: ‘updated_at’.

image

Something is surely broken

Update 4: We had to recreate our “assistant” and disbale code interpreter. For some reason the code interpreter was launching during dialogie, and once code-interpreter was launched it seems that it would not use actual functions, instead it would only “mock” results. This was a substantial change in operation affecting multiple models. It is disappointing that it was not communicated, but I am sharing it here in case anyone else has this issue.

2 Likes
2

Update: I checked back our server logs. It was working fine until yesterday. And all of a sudden today, with no changes, it refuses to call any functions. It only hallucinates the response and provides it. And it often says there has been a “slight hiccup” in running the function, except when we see our logs it is not actually calling a tool/function at all something got broken bad.

2 Likes
3

Update 2: We have not tested with GPT4; GPT-3.5-Turbo (the new ones); and both the November and January GPT4-Turbo models.

ALL mdoels have stopped calling functions/tools even when explicitly instructed to do so.

4

I looked for any changes in my previous case of tools being used when absolutely not useful.

Answer: -0125 is still broken.

system: you are chatgpt…
functions: get_random_float, get_random_int
user: What is the capital of France? What is the capital of Germany?

stupidity comes out:

  "object": "chat.completion",
  "created": 1707797239,
  "model": "gpt-3.5-turbo-0125",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "id": "call_Z...",
            "type": "function",
            "function": {
              "name": "get_random_int",
              "arguments": "{\"range_start\": 1, \"range_end\": 3}"
            }
          },
          {
            "id": "call_1...",
            "type": "function",
            "function": {
              "name": "get_random_int",
              "arguments": "{\"range_start\": 1, \"range_end\": 3}"
            }
          }
        ]
      },
      "logprobs": {
        "content": null
      },
      "finish_reason": "tool_calls"

on chat completions where you get to see what you send. Or re-engineer function calling to see the root invocation of the dumbnity.

[{"full_tool_call":"multi_tool_use.parallel","parameters":{"tool_uses":[{"recipient_name":"functions.get_random_int","parameters":{"range_start":1,"range_end":100}},{"recipient_name":"functions.get_random_int","parameters":{"range_start":1,"range_end":100}}]}

5

Update 3: When trying to duplicate Assistants to test, you now get an error "unknown parameter: ‘updated_at’.

image

Something is surely broken.

6

I was going to say “ask the AI what # Tools and what ## functions it has available”, but the last reply speaks to more problems.

1 Like
7

indeed. It’s straight up mocking results. See snip from Playground:

image
image967×213 7.61 KB

2 Likes
8

It seems the AI has emitted to code interpreter. Calling tools, or at least the start tokens of tool calls from which the AI cannot escape, when it is not needed or wanted, is what I describe, and in this case, it decided to send to python.

9

Is anyone else having the issue with function calls suddenly not running? Or Mocked results instead of tool calls?

The model is now openly responding by saying thay in a live environment they would query the API, but that it cannot do that in its current configuration, and so it is “simulating a basic response”.

??

10

You are right. The issue is that once code interpreter commences, it cannot seems to exit that and go back to function calling.

11

Another thing: functions should not be asked for by the user. They should be fully described and instead fulfill a user need.

Use system prompts about them, or talk about them, and you are more likely to get an AI that writes code instead of using functions correctly when “user typed the word functions = first likely token is python”.

12

Good tip. Thank you fellow traveler :slight_smile: