I review the documentation for providers that we use periodically, and I came across this helpful snippet in Anthropic’s tool use documentation:
When you call the Anthropic API with the tools parameter, we construct a special system prompt from the tool definitions, tool configuration, and any user-specified system prompt. The constructed prompt is designed to instruct the model to use the specified tool(s) and provide the necessary context for the tool to operate properly:
In this environment you have access to a set of tools you can use to answer the user's question.
{{ FORMATTING INSTRUCTIONS }}
String and scalar parameters should be specified as is, while lists and objects should use JSON format. Note that spaces for string values are not stripped. The output is not expected to be valid XML and is parsed with regular expressions.
Here are the functions available in JSONSchema format:
{{ TOOL DEFINITIONS IN JSON SCHEMA }}
{{ USER SYSTEM PROMPT }}
{{ TOOL CONFIGURATION }}
I assume OpenAI does something similar. It would be helpful if you provided the structure of your tool usage prompt in the documentation. I’ve experimented with hundreds of different formats for tool calls, changing parameter order, description formatting, anything you can think of. I can confirm that formatting is a factor in model performance for calling tools. Knowing the format that already exists in the behind-the-scenes prompt would allow us to craft schema that is faster and more accurate for us, and less demanding of OpenAI compute resources.
To be clear, are you suggesting using structured outputs instead of tool calls, or something else? I suppose that’s all they are behind the scenes, but it is exhaustive to constantly reinvent the wheel. It’s ridiculous.
The total lack of support even on these forums has demolished my organization’s confidence in their platform. Once our service has stabilized, one of the first upgrades will be to roll out our own small model and cut all dependencies on OpenAI, Anthropic, and whatever other fallbacks we use in the meanwhile.
I would ditch structured outputs and potentially json mode too.
I use none of that stuff anywhere, and it gives you a clean, vendor agnostic model adapter (with some minor caveats).
The models are plenty capable of abiding by a (reasonable) schema. Have been since davinci-002.
I always argue that if a model can’t abide by the schema (i.e. parsing error) it’s a good indicator that it wouldn’t really understand the prompt either. But it depends on what you’re doing, ofc.
Ah, I wouldn’t throw all the babies out with the bathwater. OAI has some redeeming qualities. The “base” models are pretty good.
I don’t really care for all the vendor lock-in stuff (assistants, tools, structured outputs, their caching scheme, o1, etc.) If you ignore all that, it’s pretty stable. No wheel reinventing required. And everything becomes quite portable too.
4o-mini offers by far the best cost/capability ratio of anything I’ve used so far. It legitimately opened up new possibilities for AI-supported applications. I don’t know how long it will last - no company likes to have loss leaders in the lineup forever - but it is the reason I have stuck around and put up with the black boxes.
The only “tool” that you have access to is a functions tool, for function-calling.
OpenAI reserves their actual higher level tools (and a named API backend recipient for catching triggered responses to them) for themselves.
The models are post-trained on function-calling, so the transformation of a function specification into AI language being placed, a heading “# Tools” after your system prompt language with a function tool, is not directly instructed but rather is understood by the AI model in the structure used.
The only useful thing to know is when the function fails to be understood, such as capricious use of the ‘required’ signal, or convoluted object or array nesting that a human would have a hard time deciphering.
You can make a system prompt giving you developer authority to have the # Tools section dumped out completely.
Are we certain of this? Different implementations are certainly possible, but I would be surprised if there was such a fundamental difference between OpenAI and Anthropic.
Can you provide an example? I’m having difficulty getting it to return a reasoning message alongside the tool request message. Or do you achieve this by including a “reasoning” parameter in each tool?