Model Skips Phone Number Validation and Auto-Corrects Input to Pass Validation Rules

API
1

Hi everyone,

I’m currently building a real-time AI voice assistant using OpenAI’s real-time voice API. The assistant reads out questions from a structured JSON, asks them to the caller, and is expected to collect and validate responses based on rules also defined in that JSON.

Issue Summary:

We’re facing major issues with the validation logic not being followed consistently. Specifically:

  1. When a user provides too few or too many digits, the model sometimes skips validation entirely and proceeds to the next question.
  2. In some cases, it prompts the user to retry once or twice, but even if the user gives an invalid response, it still accepts it and moves on.
  3. Worse, the model occasionally auto-corrects the input by either truncating or appending digits to match the expected format — which we do not want. We need strict validation only, with no guessing or fixing, this is happening with all other validations like (credit card number, postalcode, etc).

Current Implementation:

  • A master prompt is defined in session.update, contains general behavior guidelines and validation responsibilities for the model.
  • Element-specific prompts are passed via conversation.item.create, including detailed instructions for each type of question (phone number, credit card, etc.) along with their associated validation rules in JSON.

Tried Solutions:

  1. Refactored prompts multiple times (short, long, structured, etc.).
  2. Validated against multiple models including (gpt-4o-realtime-preview-2025-06-03, gpt-4o-realtime-preview-2024-12-17).
  3. Tried putting validation instructions in different places (session.update, conversation.item.create, response.create).
  4. Added strict language in prompt to instruct the model to never modify or assume values.

What We Need:

  • Is there a way to force the model to perform strict validation without attempting to correct or fix user input?
  • Are there known prompt patterns or API strategies to prevent the model from skipping validation?

Please check this prompt and json.

#### Check `ValidationRules`Field if Not Empty:
- Before reading out the question, read the expected input format based on the `ValidationRules` provided.
- After collecting the user response, strictly validate the response against the `ValidationRules`.
- Do not assume, auto-correct, or modify the user's input.
- On invalid input:
    - Say: ""The response provided does not meet the validation criteria. Please try again and ensure the input matches the required format.""
    - Keep retrying until valid response is received.

%{
  "item" => %{
    "content" => [
      %{"text" => "", "type" => "input_text"},
      %{
        "inputItems" => [
          %{
            "Description" => "Can I have your phone number, please?",
            "Required" => true,
            "ValidationRules" => "Phone number must be exactly 7 digits"
         }
        ]
      }
    ],
    "role" => "user",
    "type" => "message"
  },
  "type" => "conversation.item.create"
}

Any insights or suggestions would be greatly appreciated!

2

At some point, the LLM is being fed so many scripted answers that you can just optimize away the entire thing with a structured web form.

As a sidenote, the models are primarily trained in natural language. Many of your instructions appear to be delivered in code or some schema format. This isn’t really an ideal way to prompt the model.