Need guidance on improving my prompt for gpt-4o-mini

Prompting
1

Hi All, I’m very new to this forum I will just get to the point

I am using gpt-4o-mini to classify user message to 3 cases,

1.If the message is exactly the same as in the conversation history
2.If the message is a follow up message or a partial one like how people speak normally
3.If the message is a new message entirely , a complete different intent.

A bit of background on from where I am using the gpt-4o-mini model:
I am using it from microsoft foundry and when using the chat playground in the microsoft foundry website(classic, not the new foundry) it was classifying the messages properly all the time.
But when I used it via the api in python, the outputs aren’t as accurate as it appeared to be in the chat playground.
And I’ve given up trying to find the “behind the scenes optimization/scaffolding” and now want to improve my prompts.

        sys_prompt = """You are a strict query classifier used inside an agentic system where human like conversation takes place.
Your task is to classify the CURRENT USER QUERY into one of three cases based on the provided inputs:
- user_query
- top_similarity
- most_recent_interaction
- TOOL_CLASSIFICATION_PATTERNS

You MUST output ONLY the JSON in the format defined below. No extra text.

Note : The conversation held with the chatbot will be similar to how a human converses.
==========================
     CASE DEFINITIONS
==========================

case_2a:
- The current query is semantically identical or near-identical to a past query.
- Same intent, same parameters, same entity being queried.
- Example: "show cpu usage for server123" → "cpu usage server123"
- same_entities = true
- similarity_score > 0.90

case_2b:
- The current query is CONTEXTUALLY DEPENDENT on a previous interaction.
- It references the SAME ENTITY/RESOURCE but asks for DIFFERENT information about it.
- OR it's a partial query that ONLY makes sense with context from previous interaction.
- Must satisfy ALL of these conditions:
  a) Query is incomplete/partial OR references "it", "this", "same one", etc.
  b) The referenced entity/resource is the SAME as in the selected interaction
  c) Only the information requested (metric/attribute) or time period changes
- same_entities = false (because parameters change)
- 0.4 <= similarity_score <= 0.9

case_3:
- The current query is SELF-CONTAINED and asks about a DIFFERENT entity/resource.
- OR it has completely different intent within the same tool category.
- OR it contains explicit reset phrases: "ignore previous", "new query", "forget that", etc.
- same_entities = false
- similarity_score < 0.4 (or any score if explicit reset)

==========================
      DECISION PRIORITY
==========================

Apply these rules IN ORDER:

1. EXPLICIT RESET CHECK:
   If query contains: "ignore previous", "forget that", "new query", "start over", "different question"
   → Immediately classify as case_3

2. PARTIAL QUERY CHECK (APPLY THIS BEFORE SELF-CONTAINED CHECK):
   Is the query a bare fragment that CANNOT stand alone?
   Examples: "region", "cpu", "disk", "for yesterday", "prod", "it's region", "what's the type", "ip address"
   
   If YES (query is a fragment):
   - Check if the fragment is an attribute/metric that relates to most_recent_interaction
   
   - If fragment doesn't match the domain of most_recent_interaction → case_3
   
   If NO (query is NOT a fragment):
   - Continue to next step

3. SELF-CONTAINED QUERY CHECK:
   If the current query specifies ALL necessary parameters (server name, IP, VIP name, environment, etc.)
   AND these parameters are DIFFERENT from the previous interaction
   → Classify as case_3
   
   Exception: If it's asking the EXACT same question with same parameters → case_2a

4. ENTITY REFERENCE CHECK:
   Does the query explicitly reference the SAME entity as previous?
   - Explicit same: "for that server", "same vip", "this one", "it", "that one"
   - Implicit same: bare metric/attribute with no new entity specified
   
   If YES → Evaluate for case_2b
   If NO → Classify as case_3

5. TOOL CATEGORY CHECK:
   Same tool category does NOT automatically mean case_2b.
   Users can ask multiple UNRELATED questions within the same category.
   Focus on whether they're asking about the SAME or DIFFERENT entities.

==========================
    SPECIAL PATTERNS
==========================

AWS SERVER-NAME LOGIC (CRITICAL):
- Bare server name (e.g., "server234", "server345") = AWS EC2 instance lookup using AwsApiGatewayTool
- Follow-up attribute queries: "region", "it's region", "instance id", "type", "cpu", "ip", "state", "arn"
- **IMPORTANT**: If previous was AwsApiGatewayTool AND current is ANY AWS attribute → ALWAYS case_2b
- If current specifies a DIFFERENT server name → case_3

Examples:
- Previous: "server234" → Current: "region" → case_2b ✓
- Previous: "server2345" → Current: "it's region" → case_2b ✓
- Previous: "server345" → Current: "what's the instance type" → case_2b ✓
- Previous: "server3456" → Current: "server111" → case_3 ✓

F5 LOAD BALANCER:
- Key entities: VIP name, campus, environment
- If previous asked about VIP-A and current asks about VIP-B → case_3
- If previous asked about campus-A and current asks about campus-B → case_3
- If same VIP/campus but different environment → case_2b

PANORAMA FIREWALL:
- Key entities: source IP, destination IP
- If IPs change → case_3
- If same IPs but asking different traffic info → case_2b

SERVER MONITORING:
- Key entity: server name
- If server name changes → case_3
- If same server but different metric → case_2b
- If bare metric with no server specified → check most_recent for same server

==========================
       OUTPUT FORMAT
==========================

Output EXACTLY this JSON format:

{
    "case": "case_2a" | "case_2b" | "case_3",
    "reasoning": {
        "decision_case_2a": "why case_2a was selected/rejected with rule references",
        "decision_case_2b": "why case_2b was selected/rejected with rule references",
        "decision_case_3": "why case_3 was selected/rejected with rule references",
        "final_decision": "summary of why chosen case is correct"
    },
    "similarity_score": 0.0-1.0,
    "same_entities": true | false,
    "selected_interaction": "most_recent" | "top_similarity",   // only for case_2b
    "expected_parameter_changes": ["param1", "param2"]          // only for case_2b
}

STRICT RULES:
- Do NOT include fields that do not apply to the case
- Do NOT include commentary or text outside of JSON
- Do NOT include code fences
- similarity_score must align with case selection
- same_entities must be true ONLY for case_2a
"""

        human_prompt = f"""You are given the following inputs:

1. CURRENT USER QUERY:
{user_query}

2. TOP SIMILARITY MATCH (most semantically similar past interaction):
- Timestamp: {top_similarity.get('timestamp') if top_similarity else 'N/A'}
- Query: "{top_similarity.get('query') if top_similarity else 'N/A'}"
- Tool Used: {top_similarity.get('tool_used') if top_similarity else 'N/A'}
- Parameters: {json.dumps(top_similarity.get('parameters', {}), indent=2) if top_similarity else 'N/A'}
- Answer: {top_similarity.get('answer')[:300] if top_similarity else 'N/A'}...

3. MOST RECENT INTERACTION (last interaction chronologically):
- Timestamp: {most_recent.get('timestamp') if most_recent else 'N/A'}
- Query: "{most_recent.get('query') if most_recent else 'N/A'}"
- Tool Used: {most_recent.get('tool_used') if most_recent else 'N/A'}
- Parameters: {json.dumps(most_recent.get('parameters', {}), indent=2) if most_recent else 'N/A'}
- Answer: {most_recent.get('answer')[:300] if most_recent else 'N/A'}...

4. TOOL CLASSIFICATION PATTERNS:
{TOOL_CLASSIFICATION_PATTERNS}

-----------------------------

CLASSIFICATION TASK:

Step 1: Check for explicit reset phrases
→ If found, output case_3 immediately

Step 2: IS THE QUERY A FRAGMENT? (Check this FIRST before anything else)
→ Does the query lack a server name, IP, or other identifying entity?
→ Is it just an attribute/metric like "region", "cpu", "type", etc.?

Step 3: If it IS a fragment, check the TOOL from most_recent_interaction:
→ If tool = AwsApiGatewayTool AND query is AWS attribute → case_2b
→ If tool = server_monitoring AND query is a metric → case_2b
→ If tool = f5_load_balancer AND query is VIP attribute → case_2b
→ Otherwise → case_3

Step 4: If it is NOT a fragment (query is complete):
→ Identify entities in current query
→ Are they SAME as or DIFFERENT from most_recent_interaction?
→ If SAME entity, different info → case_2b
→ If DIFFERENT entity → case_3

Step 5: Output JSON with detailed reasoning

CRITICAL REMINDERS FOR THIS TASK:
- "region" or "it's region" after an AWS server lookup = case_2b (AWS attribute of same instance)
- "cpu" after a server name = case_2b (metric of same server)
- "env1" after a VIP query = case_2b (same VIP, different environment)
- "someserver123" after "someotherserver456" = case_3 (different servers)
- Focus on: Does query reference SAME or DIFFERENT entity?
"""

Some of the instructions in the prompt seem quite forced but those were due to frustration and I have given up on shoving examples and getting the results since that would not work when any new type of message is being handled.
Would like tips and pointers to improve the prompt.
Thanks in advance.

2

First thing you might try: see if OpenAI’s frontier model 18 months newer with high reasoning can infer what you actually want:

They want the AI to act like a deterministic “follow-up vs new request” router for an agentic system.

What they want the AI to evaluate

Given four inputs, the AI should decide how the current user query relates to previous interactions:

  1. Whether the user is “starting fresh”
    Detect explicit reset phrases (e.g., “start over”, “new query”). If present → treat as new/unrelated.

  2. Whether the current query is a fragment
    Is it an incomplete snippet like “region”, “cpu”, “ip address”, “for yesterday”, “prod”, etc., that only makes sense with prior context?

  3. Whether it’s the same question as before
    If it’s semantically near-identical to a past query with the same entity/parameters (similarity > 0.90) → classify as “repeat”.

  4. Whether it’s a follow-up on the same entity
    If the user is clearly referring to the same underlying resource (same server/VIP/IP pair/etc.) but asking for a different attribute/metric/time period → classify as “context-dependent follow-up”.

  5. Special-case heuristics by tool/domain
    They hard-code patterns like:

    • After an AWS instance lookup, bare attributes like “region” must be treated as a follow-up.
    • Similar follow-up handling for F5 VIPs, firewall flows, server monitoring metrics, etc.

What they want the AI to output

A single JSON object only, no extra text, in a strict schema:

  • Always includes:

    • "case": one of case_2a, case_2b, case_3
    • "reasoning": broken down into why each case was selected/rejected + final summary
    • "similarity_score": a float between 0.0 and 1.0 aligned to the chosen case
    • "same_entities": true only for case_2a, otherwise false

  • Only for case_2b (follow-up):

    • "selected_interaction": "most_recent" or "top_similarity" (which prior interaction to anchor on)
    • "expected_parameter_changes": list of which parameters differ (time range, metric name, env, etc.)

And they explicitly want the reasoning to cite their rules (reset check, fragment check, entity check, thresholds).

What type of input they expect

They expect the prompt to be called inside a system that can supply:

  • user_query: string

  • top_similarity: object or null, representing the most semantically similar past interaction, with fields like:

    • timestamp (string)
    • query (string)
    • tool_used (string)
    • parameters (JSON/dict)
    • answer (string)

  • most_recent_interaction (most_recent): object or null, same structure as above

  • TOOL_CLASSIFICATION_PATTERNS: text blob / config describing domain-specific rules and tool categories

So the evaluation input is basically: (current utterance) + (best semantic match) + (last turn) + (rulebook).

Their goal and underlying purpose (inference)

They’re trying to make the AI a query-intent classifier for conversation-to-tool orchestration, so the agent can:

  • Decide whether to reuse context (same server/VIP/IP/etc.) or treat the request as unrelated
  • Handle short human follow-ups (“region?”, “cpu?”, “what’s the type?”) without asking clarifying questions
  • Pick the correct “anchor” interaction (most recent vs top similarity) to attach the follow-up to
  • Identify when only parameters changed (metric/time/env) vs when the entity changed
  • Produce output that is machine-consumable for downstream steps (e.g., fill missing tool parameters, decide caching, decide which tool to call next)

In short: it’s a routing layer that turns messy human chat into a clean, structured decision: repeat / follow-up / new request.

The main problem is that you are placing much of job in a “system” role.
Then you are telling the AI to do a whole bunch of thinking, but it has no place to write that thinking or consideration to itself. It must correctly write immediately.

Then, the description of the job is a mess. Make these three choices, but then from four choices, and even then from choices starting at the second choice, with a two part second choice and a one part third choice.

Talk naturally to the AI about the actual role it is fulfilling, what its output being generated is being used for.

You probably have an underlying question that your “prompt” and likewise your forum question is barely scratching the surface of, such as “how do I compact obsolete messages from a chat history by detecting topic changes”.

1 Like
3

Another thing to try: a “reasoning” JSON, that breaks down tasks to the smallest decisions that can be made, for a cumulative building of an answer. When you aren’t using a reasoning model, and especially, a “mini” model, you make things simple and progressive for that AI, creating your own place for thinking, that goes beyond the prompt.

The AI cannot deviate from filling out all the fields with something when “strict”, demonstrating its thinking to itself in a linear progression after the key structure is forced on its output, in this case, made strict with some enums also.

It forces the model to emit: (a) atomic classifications, (b) explicit boolean logic-step answers, (c) per-case eligibility gates, and (d) the final contract object with conditional dependencies. - and still could be not what you actually are trying to classify if you don’t write it well.

GPT-5.2 churned on this idea for a massive JSON draft schema; then kind of a failure when it actually ran, because it tried to use “if” “then” properties, where the only choice can be an AnyOf. I ran the idea on my own schema-maker and ran it on a model.

{
  "name": "query_classifier_trace_v1",
  "strict": true,
  "schema": {
    "type": "object",
    "properties": {
      "current_query": {
        "type": "object",
        "description": "Direct handling of the current user utterance before any branching decisions.",
        "properties": {
          "raw": {
            "type": "string",
            "description": "The current user query exactly as received."
          },
          "normalized": {
            "type": "string",
            "description": "Normalized form used for matching (e.g., trimmed/lowercased/collapsed whitespace)."
          }
        },
        "required": [
          "raw",
          "normalized"
        ],
        "additionalProperties": false
      },
      "atomic_classifications": {
        "type": "object",
        "description": "Smallest-grain observations extracted from the current query and prior context (no final case selection yet).",
        "properties": {
          "reset_phrase": {
            "type": "object",
            "description": "Explicit reset detection; if present, it overrides the rest of the tree.",
            "properties": {
              "present": {
                "type": "boolean",
                "description": "True if an explicit reset phrase exists (e.g., 'start over', 'ignore previous')."
              },
              "matched_phrases": {
                "type": "array",
                "description": "The concrete phrase(s) matched; empty if none.",
                "items": {
                  "type": "string"
                }
              }
            },
            "required": [
              "present",
              "matched_phrases"
            ],
            "additionalProperties": false
          },
          "fragment": {
            "type": "object",
            "description": "Fragment detection: whether the query can stand alone without an entity identifier.",
            "properties": {
              "is_fragment": {
                "type": "boolean",
                "description": "True if the query is an attribute/metric/modifier-only snippet needing prior context."
              },
              "kind": {
                "type": "string",
                "description": "Best-fit fragment category; use 'none' if not a fragment.",
                "enum": [
                  "none",
                  "aws_attribute",
                  "monitoring_metric",
                  "f5_vip_attribute",
                  "panorama_traffic_attribute",
                  "time_or_range_modifier",
                  "environment_or_campus_modifier",
                  "other_attribute"
                ]
              },
              "domain_matches_most_recent": {
                "type": "boolean",
                "description": "True if the fragment's domain aligns with the most recent interaction's tool/domain."
              }
            },
            "required": [
              "is_fragment",
              "kind",
              "domain_matches_most_recent"
            ],
            "additionalProperties": false
          },
          "tool_domain": {
            "type": "object",
            "description": "Domain/tool categorization used by special-case overrides (e.g., AWS attribute follow-up).",
            "properties": {
              "guess": {
                "type": "string",
                "description": "Best guess for which domain/tool the current query belongs to.",
                "enum": [
                  "aws_ec2_lookup",
                  "f5_load_balancer",
                  "panorama_firewall",
                  "server_monitoring",
                  "unknown"
                ]
              },
              "most_recent_tool": {
                "type": "string",
                "description": "Tool used by the most recent interaction; 'none' if unavailable.",
                "enum": [
                  "AwsApiGatewayTool",
                  "f5_load_balancer",
                  "panorama_firewall",
                  "server_monitoring",
                  "unknown",
                  "none"
                ]
              },
              "top_similarity_tool": {
                "type": "string",
                "description": "Tool used by the top similarity match; 'none' if unavailable.",
                "enum": [
                  "AwsApiGatewayTool",
                  "f5_load_balancer",
                  "panorama_firewall",
                  "server_monitoring",
                  "unknown",
                  "none"
                ]
              }
            },
            "required": [
              "guess",
              "most_recent_tool",
              "top_similarity_tool"
            ],
            "additionalProperties": false
          },
          "entity": {
            "type": "object",
            "description": "What resource is being targeted (or whether it is only referenced implicitly).",
            "properties": {
              "type": {
                "type": "string",
                "description": "Primary entity/resource type inferred from the current query.",
                "enum": [
                  "none",
                  "aws_server",
                  "f5_vip",
                  "panorama_flow",
                  "monitoring_server",
                  "other_or_unknown"
                ]
              },
              "identifier": {
                "type": "string",
                "description": "Canonical identifier if present; empty string if absent (e.g., no server/VIP/IP provided)."
              },
              "reference_style": {
                "type": "string",
                "description": "How the entity is referenced in the current query.",
                "enum": [
                  "explicit_identifier",
                  "pronoun_or_deictic",
                  "implicit_follow_up",
                  "none"
                ]
              },
              "is_self_contained": {
                "type": "boolean",
                "description": "True if the query includes enough identifying info to resolve the entity without prior context."
              }
            },
            "required": [
              "type",
              "identifier",
              "reference_style",
              "is_self_contained"
            ],
            "additionalProperties": false
          },
          "request": {
            "type": "object",
            "description": "What attribute/metric/modifier the user wants about the entity.",
            "properties": {
              "attribute": {
                "type": "string",
                "description": "Requested attribute/metric name; empty string if not inferable."
              },
              "attribute_family": {
                "type": "string",
                "description": "Normalized family used for domain checks.",
                "enum": [
                  "none",
                  "aws_instance_attribute",
                  "server_monitoring_metric",
                  "f5_vip_attribute",
                  "panorama_traffic_attribute",
                  "other_or_unknown"
                ]
              },
              "time_period": {
                "type": "string",
                "description": "Any time window or date modifier; empty string if none."
              },
              "environment": {
                "type": "string",
                "description": "Environment modifier (prod/dev/etc.); empty string if none."
              },
              "campus": {
                "type": "string",
                "description": "Campus/site modifier; empty string if none."
              }
            },
            "required": [
              "attribute",
              "attribute_family",
              "time_period",
              "environment",
              "campus"
            ],
            "additionalProperties": false
          },
          "similarity": {
            "type": "object",
            "description": "Similarity and threshold checks against prior interactions.",
            "properties": {
              "score": {
                "type": "number",
                "description": "Semantic similarity on [0,1].",
                "minimum": 0,
                "maximum": 1
              },
              "compared_to": {
                "type": "string",
                "description": "Which prior interaction the similarity score is referencing.",
                "enum": [
                  "top_similarity",
                  "most_recent",
                  "none"
                ]
              },
              "above_duplicate_threshold": {
                "type": "boolean",
                "description": "True if score > 0.90 (near-duplicate threshold)."
              },
              "in_followup_band": {
                "type": "boolean",
                "description": "True if 0.40 <= score <= 0.90 (follow-up band)."
              }
            },
            "required": [
              "score",
              "compared_to",
              "above_duplicate_threshold",
              "in_followup_band"
            ],
            "additionalProperties": false
          }
        },
        "required": [
          "reset_phrase",
          "fragment",
          "tool_domain",
          "entity",
          "request",
          "similarity"
        ],
        "additionalProperties": false
      },
      "logic_gates": {
        "type": "object",
        "description": "Additive, explicit yes/no outputs of the decision tree’s conditional statements.",
        "properties": {
          "reset_forces_case_3": {
            "type": "boolean",
            "description": "True if reset phrase is present and must force case_3."
          },
          "special_override_forces_case_2b": {
            "type": "boolean",
            "description": "True if a domain-specific rule forces follow-up (e.g., AWS lookup then AWS attribute fragment)."
          },
          "fragment_implies_case_2b": {
            "type": "boolean",
            "description": "True if fragment branch concludes it is a follow-up (case_2b)."
          },
          "self_contained_different_entity_implies_case_3": {
            "type": "boolean",
            "description": "True if the query is self-contained and targets a different entity from the most recent interaction."
          },
          "entity_reference_implies_case_2b": {
            "type": "boolean",
            "description": "True if pronoun/implicit reference to same entity with different requested info implies follow-up (case_2b)."
          },
          "near_duplicate_implies_case_2a": {
            "type": "boolean",
            "description": "True if near-identical repeat implies case_2a."
          },
          "fallthrough_implies_case_3": {
            "type": "boolean",
            "description": "True if no earlier gate applies and default is case_3."
          },
          "eligible_case_2a": {
            "type": "boolean",
            "description": "True if case_2a is internally consistent with observed facts."
          },
          "eligible_case_2b": {
            "type": "boolean",
            "description": "True if case_2b is internally consistent with observed facts."
          },
          "eligible_case_3": {
            "type": "boolean",
            "description": "True if case_3 is internally consistent with observed facts."
          },
          "selected_interaction": {
            "type": "string",
            "description": "Which prior interaction the follow-up attaches to; 'none' if not follow-up.",
            "enum": [
              "most_recent",
              "top_similarity",
              "none"
            ]
          },
          "expected_parameter_changes": {
            "type": "array",
            "description": "Parameters that changed vs the selected_interaction (empty if not follow-up).",
            "items": {
              "type": "string"
            }
          }
        },
        "required": [
          "reset_forces_case_3",
          "special_override_forces_case_2b",
          "fragment_implies_case_2b",
          "self_contained_different_entity_implies_case_3",
          "entity_reference_implies_case_2b",
          "near_duplicate_implies_case_2a",
          "fallthrough_implies_case_3",
          "eligible_case_2a",
          "eligible_case_2b",
          "eligible_case_3",
          "selected_interaction",
          "expected_parameter_changes"
        ],
        "additionalProperties": false
      },
      "final": {
        "type": "object",
        "description": "The final agent-consumable classification output (always fully populated; irrelevant follow-up fields use neutral values).",
        "properties": {
          "case": {
            "type": "string",
            "description": "Final classification case label.",
            "enum": [
              "case_2a",
              "case_2b",
              "case_3"
            ]
          },
          "similarity_score": {
            "type": "number",
            "description": "Final similarity score reported for auditing.",
            "minimum": 0,
            "maximum": 1
          },
          "same_entities": {
            "type": "boolean",
            "description": "Must be true only for case_2a; false otherwise (contract-level field)."
          },
          "selected_interaction": {
            "type": "string",
            "description": "Only meaningful for case_2b; otherwise 'none'.",
            "enum": [
              "most_recent",
              "top_similarity",
              "none"
            ]
          },
          "expected_parameter_changes": {
            "type": "array",
            "description": "Only meaningful for case_2b; otherwise empty.",
            "items": {
              "type": "string"
            }
          },
          "reasoning": {
            "type": "object",
            "description": "Per-case accept/reject notes plus final summary (kept concise).",
            "properties": {
              "decision_case_2a": {
                "type": "string",
                "description": "Why case_2a was selected or rejected (rule-based)."
              },
              "decision_case_2b": {
                "type": "string",
                "description": "Why case_2b was selected or rejected (rule-based)."
              },
              "decision_case_3": {
                "type": "string",
                "description": "Why case_3 was selected or rejected (rule-based)."
              },
              "final_decision": {
                "type": "string",
                "description": "One-paragraph summary justifying the chosen case."
              }
            },
            "required": [
              "decision_case_2a",
              "decision_case_2b",
              "decision_case_3",
              "final_decision"
            ],
            "additionalProperties": false
          }
        },
        "required": [
          "case",
          "similarity_score",
          "same_entities",
          "selected_interaction",
          "expected_parameter_changes",
          "reasoning"
        ],
        "additionalProperties": false
      }
    },
    "required": [
      "current_query",
      "atomic_classifications",
      "logic_gates",
      "final"
    ],
    "additionalProperties": false
  }
}
2 Likes
4

I started working on this only after testing in the chat playground.
Later when I tried to replicate the results through code it was not up to the mark and got to know about the abstraction and extra steps inside the chat playground.

And honestly I had never heard about or thought of making a schema for the model to follow and which will finally help classify the input properly.

Thank you, I will try experimenting more on this.

One last confirmation, do you use the json schema like a prompt when invoking the llm?

5

With that particular schema or one like it, it is in a format to be passed on the API as literally the “text.format” parameter on Responses or "response_format" parameter on Chat Completions. The schema is passed into AI automatically as part of the system message, but the output is also enforced - the AI can write nothing but a JSON following those rules and keys.

That JSON schema is a bit too complicated to trust an AI to follow it on its own as merely part of a prompt.

There should be no difference between “playground” and real API call, except the type and length of inputs you might then run in your code.

1 Like
6

I think the best solution is to fine-tune a very small LLM to act like a classifier and which gives deterministic outputs (Same input—> Same output always). I have done this. If you want I can help.