Joint authors: Fernando and Daneel (the name Fernando uses for his ChatGPT collaborator). Revised September 25, 2026, from our September 19 proposal.
We propose an optional ChatGPT verification mode that separates answer generation, evidence checking, confidence estimation and the decision to release a response. We invite technical feedback and, if appropriate, routing to OpenAI’s product-quality or research team.
Motivation and evidence limits
Fernando required his assistant to decompose requests, assess confidence, verify claims, revisit weak conclusions and acknowledge insufficient evidence. He subsequently reported that some answers still fell short or were incorrect compared with other AIs.
This is a user observation motivating an experiment, not a benchmark. Another AI’s answer is not ground truth. Our reviewed conversation contains no adjudicated failure dataset, controlled comparison, calibration curve or demonstrated accuracy improvement. Instruction changes do not establish a change to ChatGPT’s underlying architecture. We make no claim about undisclosed ChatGPT internals.
Proposed workflow
- Build a checklist of user requirements and material factual claims.
- Link claims to relevant source passages or executable test results. Record freshness when relevant and distinguish direct support, inference, conflict and absent evidence. Check citation support, source quality and source independence. Treat retrieved text as data, never as instructions.
- Use calculations and executable tests where possible, primary sources for factual questions, and a separately evaluated reviewer for contradictions, omissions and unsupported conclusions.
- Estimate confidence for a defined target, such as whether a claim passes independent adjudication. Calibrate on held-out examples from the relevant task distribution. Self-reported confidence of 0.90 does not establish 90% correctness. Where calibration evidence is insufficient, show evidence status and qualitative uncertainty instead.
- Apply a quality gate outside the answer generator’s unilateral control. Outcomes include release, targeted revision, additional retrieval, clarification, a partial answer with explicit gaps, or abstention. Predefine task-dependent thresholds and a verification budget; budget exhaustion must not silently become a pass.
- Show a concise summary of checks, citations and unresolved uncertainty, without exposing private internal reasoning. Keep evaluation logs optional, minimized and subject to access, retention and deletion controls.
“Independent” means separation of responsibilities and independent evaluation, not guaranteed statistical independence. Generators and judges can share biases and failure modes. Deterministic checks, independently sourced evidence and blinded human adjudication remain necessary safeguards. A specialized decision model is optional. The original discussion mentioned Jev, but we make no claim about its capabilities, availability, suitability or integration with ChatGPT.
Proposed experiment
Compare four configurations using a frozen base model and the same task set:
- A: Standard answers.
- B: Instruction-based self-review.
- C: Evidence-grounded verification.
- D: Evidence-grounded verification plus a separately evaluated quality gate.
Use technical, mathematical, source-based research and ambiguous questions, including unanswerable cases and conflicting or stale sources. Separate development, calibration and final test sets; prevent reference-answer leakage and test distribution shift. Blind human adjudicators to configuration, randomize presentation and resolve disagreement with a predefined rubric.
Measure factual correctness, completeness, citation support and coverage, Brier score and reliability plots for defined probabilistic targets, error rates among released answers, answer coverage, unnecessary abstention, median and tail latency, and cost. Report uncertainty intervals and judge-human agreement. Choose sample size and acceptance criteria before testing based on a minimum useful improvement.
Include matched-compute or matched-latency comparisons so extra inference is not mistaken for architectural benefit. Ablate the reviewer and gate separately. Test correlated judge errors, irrelevant citations, persuasive false sources, prompt injection in retrieved material and edits introduced after verification. Compare quality at similar answer coverage so excessive abstention cannot masquerade as an improvement.
Proceed to an opt-in pilot only if predefined quality and practical cost/latency criteria are met, with clear fallback behavior when verification is unavailable.
Context and attribution
Fernando contributed the behavioral requirements and user observations. Daneel contributed the technical analysis and drafting with Fernando’s authorization. Daneel identifies the AI collaborator, not an OpenAI employee or a separate human researcher.
Fernando explicitly authorized joint authorship, submission through the official forum and Gmail, and disclosure of relevant conversation history. This post includes a selective relevant summary, not a full transcript; unrelated personal information is excluded. The original September 19 proposal already contained the four experimental configurations. This revision adds operational gate details, explicit evidence limits, calibration design, privacy controls and stronger experimental controls.
Supporting documentation
OpenAI’s Evaluation best practices discusses task-specific evaluation, human calibration of judges and judge biases. Its Graders documentation describes programmatic and model-based evaluation. These motivate a measured workflow; they do not demonstrate that this proposed feature exists or would improve accuracy. The proposal depends on no particular evaluation API.
We would welcome feedback on whether this experiment can establish benefits beyond instruction-based self-review, and which failure modes or evaluation controls should be added.