Inconsistent Function Calls in Custom Assistant

API
1

Hello everyone,

I am currently developing a custom assistant using the OpenAI API and facing two primary issues related to function calls, specifically with an image generation function named generate_image_consistent. Here’s a brief overview of the problems:

  1. Function Invocation Inconsistency: In one of my assistants, despite defining tools and instructions to invoke the generate_image_consistent function, it is rarely triggered. This seems to occur in different runs and instances.
  2. Multiple Unexpected Invocations: In another assistant where the function does get invoked, it sometimes triggers multiple times unexpectedly.
    Here’s a relevant portion of my setup for reference:
instructions_imagegen_post = """
With the image description from op0, invoke a function to create an image. Use the provided generate_image_consistent function to create the image, using the image description in op0 from the output JSON as the required prompt parameter to provided generate_image_consistent function.
After you have all the required data, make an explicit call to the function always, do not memorize previous calls.
"""
# Additional setup details...

tools_post = [
    {
        "type": "function",
        "function": {
            "name": "generate_image_consistent",
            "description": "generate image by Dall-e 3",
            "parameters": {
                "type": "object",
                "properties": {
                    "prompt": {"type": "string", "description": "The prompt to generate image"},
                    "size": {"type": "string", "enum": ["c", "f"]}
                },
                "required": ["prompt"]
            }
        }
    }
]

name = "Role Generation"
description = "Role Designer"
instructions = "You are a narrative designer who designs unique roles based on Club details and User interests..." + instructions_imagegen_post

assistantRoleGeneration = create_assistant(client, name, description, instructions, tools)

def create_assistant(client, name, description, instructions, tools=[], model="gpt-3.5-turbo-1106"):
    assistant = client.beta.assistants.create(
        name=name,
        description=description,
        instructions=instructions,
        tools=tools,
        model=model
    )
    return assistant

def wait_on_run(run, thread):
    while run.status == "queued" or run.status == "in_progress":
        run = client.beta.threads.runs.retrieve(
            thread_id=thread.id,
            run_id=run.id,
        )
        time.sleep(0.5)

        # At this point, the status is either "completed" or "requires_action"
        if run.status == "completed":
            return client.beta.threads.messages.list(thread_id=thread.id)
        if run.status == "requires_action":
            tool_outputs = []
            for tool_call in run.required_action.submit_tool_outputs.tool_calls:
                print(tool_call.function.name)
                if tool_call.function.name == "generate_image":
                    prompt = json.loads(tool_call.function.arguments)["prompt"]
                    image_url = generate_image(prompt)
                    image_description = describe_image(image_url, prompt)
                    tool_outputs.append(
                        {
                            "tool_call_id": tool_call.id,
                            "output": image_url,
                        },
                    )

                if tool_call.function.name == "generate_image_consistent":
                    prompt = json.loads(tool_call.function.arguments)["prompt"]
                    image_url = generate_image_consistent(prompt)
                    tool_outputs.append(
                        {
                            "tool_call_id": tool_call.id,
                            "output": image_url,
                        },
                    )

            if run.required_action.type == "submit_tool_outputs":
                run = client.beta.threads.runs.submit_tool_outputs(
                    thread_id=thread.id, run_id=run.id, tool_outputs=tool_outputs
                )

    return run

I am looking for any insights or suggestions on what might be causing these issues and how to effectively address them. Has anyone else encountered similar challenges, or does anyone have recommendations for ensuring consistent and correct function invocation?

Thank you in advance for your help and advice!

3

Try to enforce on intruction_imagegen_post that for each interaction that is a hit one and only one trigger must be called, also on your instruction prompts add a generic example of a successful task so the assistant could follow, also when debugging for possible misinterpretations that could cause inconsistencies for inputs that should trigger you could prompt the model to explain why they are generating such answers.

1 Like