# OpenAI eval API - text similarity grading

**URL:** <https://community.openai.com/t/openai-eval-api-text-similarity-grading/1368788>\
**Category:** API\
**Tags:** evals\
**Created:** [December 9, 2025, 9:42am UTC](https://community.openai.com/t/openai-eval-api-text-similarity-grading/1368788 "2025-12-09T09:42:23Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mcling](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/mcling/32/451722_2.png) [@mcling](https://community.openai.com/u/mcling)\
**Post date:** [December 9, 2025, 9:42am UTC](https://community.openai.com/t/openai-eval-api-text-similarity-grading/1368788/1 "2025-12-09T09:42:23Z")

</div>

Hi

I have a question about using the eval API.

For the text\_similarity grader, what is the parameter in the `testing_criteria` to decide the passing grade?

Here is my code and all the results show `passing` even if the similarity is below 0.5

```auto
eval_create_result = client.evals.create(
    name="Similarity Check",
    metadata={
        "description": "This eval tests text similarity"
    },
    data_source_config={
        "type": "custom",
        "item_schema": Query.model_json_schema(), # we will upload python objects as run data
        "include_sample_schema": True
    },
    testing_criteria=[
        {
            "type": "text_similarity",
            "name": "Compare text similarity",
            "input": "{{ sample.output_text }}",
            "evaluation_metric": "cosine",
            "reference": "{{ item.text }}",
            "passing_grade": 0.8
        }
    ],
)

```

---

<div class="post-metadata">

**Author:** ![\_j](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/_j/32/766292_2.png) [@\_j](https://community.openai.com/u/_j)\
**Post date:** [December 10, 2025, 7:53am UTC](https://community.openai.com/t/openai-eval-api-text-similarity-grading/1368788/2 "2025-12-10T07:53:21Z")

</div>

You might need to review the parameter names accepted by [textSimilarityGrader](https://platform.openai.com/docs/api-reference/evals/create#evals_create-testing_criteria-textsimilaritygrader)…

```auto
{
    "type": "text_similarity",
    "name": string,
    "input": string,
    "reference": string,
    "pass_threshold": number,
    "evaluation_metric": "cosine"
}

```

`pass_threshold` is what should return a boolean.

---

<div class="post-metadata">

**Author:** ![mcling](https://sea2.discourse-cdn.com/openai1/user_avatar/community.openai.com/mcling/32/451722_2.png) [@mcling](https://community.openai.com/u/mcling)\
**Post date:** [December 10, 2025, 8:08am UTC](https://community.openai.com/t/openai-eval-api-text-similarity-grading/1368788/3 "2025-12-10T08:08:40Z")

</div>

Oh @@, thanks 🙂

Good to know the place for the documentation of the grader