Hello. I would like to request a feature: the capability and API for large language models to be able to manage described first-class objects, canonical object references, while participating in unfolding dialog, and to be able to use these objects in function calls.
By “managing object-references”, I mean obtaining and managing references to first-class objects in the form of things or values returned from previous function invocations and utilizing them as input arguments to subsequent function invocations, including after interspersed rounds of dialog or other joint tasks occurred.
As envisioned, in addition to prompts containing described functions with described parameters, prompts to large language models could come to include described first-class objects, or things.
Natural-language descriptions of those contextually-relevant objects would, ideally, allow LLMs to process natural language inputs to determine which things or objects, specifically, to provide as arguments to invoked functions in response to users’ natural-language input.
Described objects could be canonically referenced using either URIs or GUIDs. These canonical identifiers could be recognized by interoperating software applications when function calls were relayed to them.
For a concrete example, a user could engage in dialog to create four objects: a red cube, a green cube, a blue cube, and a yellow cube. Through dialog, the user could instruct an LLM to stack and unstack the cubes, referring to them in various ways.
create_cube(color) function would return a first-class object reference that could be utilized in subsequent dialog moves, subsequent function calls. The functions to move the cubes around would accept canonical object references as their arguments.
LLMs are already capable of these kinds of tasks and reasoning, as evidenced by their source-code generation functionalities. This feature request pertains to the chat completion API.
Is anyone else interested in these scenarios? Are there other approaches for developing the indicated example (which hopefully scale towards a “copilot for CAD/CAE” domain)? Any thoughts on these ideas about describing first-class objects alongside described functions with described parameters in LLM prompts? Thank you.
P.S.: See also Large Language Models.