How to create dynamic generation for Structured Outputs

I am currently using tooldantic for structured outputs in batch mode in order to produce a json output structure.

I read the following page already but I am still acing issue.
“using-pydantic-structured-outputs-in-batch-mode”

Currently I am defining the following class

from tooldantic import OpenAiResponseFormatBaseModel as BaseModel

class Scores(BaseModel):
     Score_1: int
     Score_2: int

(Note that the name of scores here are just to explain what I want to achieve)

But my issue is that depending on what the user wants sometimes that class should contain 1 score or 2 or n…

My issue is that I do not want to have n class and just pick the correct one.
How could I generate that class dynamically?

I did try modelBuilder but at runtime I then have errors

For instance

{"id": "batch_req_6723a83b7da88190a978536913103710", "custom_id": "answer-0", "response": {"status_code": 400, "request_id": "....", "body": {"error": {"message": "Invalid value: 'object'. Supported values are: 'json_object', 'json_schema', and 'text'.", "type": "invalid_request_error", "param": "response_format.type", "code": "invalid_value"}}}, "error": null}

I also did try with ‘type’ there I also have an error

from typing import List, Optional, Dict, Type

def create_score_class(scores: Dict[str, Type]):
    #Dynamically creates a class for sentence classification scores with the given score attributes.
    attributes = {score: 0 for score in scores}
    return type("SentenceClassificationScores", (BaseModel,), attributes)

myclass = create_score_class({"1":int, "2":int})

But I do have the following error:

pydantic.errors.PydanticUserError: A non-annotated attribute was detected: `1 = 0`. All model fields require a type annotation; if `1` is not meant to be a field, you may be able to resolve this error by annotating it as a `ClassVar` or updating `model_config['ignored_types']`.

So I do not succeed to have a valid dynamic json_schema.
Is there a way to do so without writing the complete json string?

Of course the prompt will also be generated accordingly.

1 Like

Anything “object” sent by pydantic BaseModel cannot grow in quantity of keys that the AI thinks of itself.

The best you can do is have an array of items with the same keys.

Here’s how you can do it:

from pydantic import BaseModel

class SingleScore(BaseModel):
    score_index: int
    score_result: str

class ScoresResponse(BaseModel):
    scores_list: list[SingleScore]

My question (with some more paste the AI could ignore)

Info:

Here is the scoring breakdown for Game 5 of the 2024 World Series between the Los Angeles Dodgers and the New York Yankees:

Team 1 2 3 4 5 6 7 8 9 Total
Dodgers 0 0 0 0 5 0 0 2 0 7
Yankees 3 1 1 0 0 1 0 0 0 6

For the Yankees, give the inning and the score only for innings where they scored.

Response:
{'scores_list': [{'score_index': 1, 'score_result': '3'}, {'score_index': 2, 'score_result': '1'}, {'score_index': 3, 'score_result': '1'}, {'score_index': 6, 'score_result': '1'}]}


Hope you can adapt that into something useful!

2 Likes

To dynamically create a Pydantic model with a variable number of fields, you need to ensure that each field is properly annotated with its type. Here’s how you can achieve this:

  1. Use a dictionary to define the fields and their types.
  2. Use the create_model function from Pydantic to dynamically create the model.

Here’s an example:

from pydantic import BaseModel, create_model
from typing import Type, Dict

def create_score_class(scores: Dict[str, Type]):
    # Dynamically creates a class for sentence classification scores with the given score attributes.
    return create_model('DynamicScores', **{score: (score_type, ...) for score, score_type in scores.items()})

# Example usage
myclass = create_score_class({"Score_1": int, "Score_2": int})
print(myclass.schema())

In this example:

  • create_score_class takes a dictionary where keys are the field names and values are the field types.
  • create_model is used to dynamically create a Pydantic model with the specified fields.

This should resolve the issue of dynamically creating a Pydantic model with a variable number of fields. :hugs:

1 Like

Hi

The code in itself is working and close to what I already tried.

but this give the following error

{"id": "batch_req_6728f54505b4819095d0c44fc85d890f", "custom_id": "answer-0", "response": {"status_code": 400, "request_id": "a7e18d855d8633b18bbec6f5fb2fed06", "body": {"error": {"message": "Invalid value: 'object'. Supported values are: 'json_object', 'json_schema', and 'text'.", "type": "invalid_request_error", "param": "response_format.type", "code": "invalid_value"}}}, "error": null}

and if I look in the produced task i have indeed a type = object

"response_format": {"properties": {"Score_1": {"title": "Score 1", "type": "integer"}, "Score_2": {"title": "Score 2", "type": "integer"}}, "required": ["Score_1", "Score_2"], "title": "DynamicScores", "type": "object"}}}

That why I would like to do the same but with tooldantic.

The error states:

Invalid value: ‘object’.

It follows up with:

Supported values are: ‘json_object’, ‘json_schema’, and ‘text’.

You can probably see what the fix to your error could be! :hugs:

The response format using this dynamic creation (with pydantic) and the one using tooldantic is totally different so even if I change the type to “json_schema” it does not work.

pydantic

{
    "response_format": {
        "properties": {
            "Score_1": {
                "title": "Score 1",
                "type": "integer"
            },
            "Score_2": {
                "title": "Score 2",
                "type": "integer"
            }
        },
        "required": [
            "Score_1",
            "Score_2"
        ],
        "title": "DynamicScores",
        "type": "json_schema"
    }
}

tooldantic (for json_schema)

{
    "response_format": {
        "type": "json_schema",
        "json_schema": {
            "name": "ScoreClass",
            "description": "",
            "strict": true,
            "schema": {
                "type": "object",
                "properties": {
                    "Score_1": {

                        "type": "integer"
                    },
                    "Score_2": {

                        "type": "integer"
                    }
                },
                "required": [
                    "Score_1",
                    "Score_2"
                ],
                "additionalProperties": false
            }
        }
    }
}

As I said I can probably create such a json string myself but I am surprised that nothing exists in tooldantic while it does in pydantic.

@j.wischnat it worked. thanks a ton. had been bashing my head on this.

for the context in my case instead of scores it was product attributes.

one heads-up (for future readers): the myclass.schema() will be deprecated and might give errors in future. instead use ‘myclass.model_json_schema’ to see the final json schema.

thanks again…

1 Like