Custom GPT → Plugins: 100-scenario baseline
OpenAI’s planned transition from Custom GPTs to Plugins raises an important question for anyone who has spent significant time refining a GPT:
How do you know whether the migrated version still behaves like the system you spent months — or years — building?
OpenAI’s official migration guidance recommends preparing existing workflows and testing the replacement Plugin before switching:
OpenAI — Custom GPT retirement and migration FAQ
Rather than migrate first and evaluate afterwards, I decided to establish a measurable baseline for my existing Custom GPT, GM INFO - IA Technologies ©, before moving it to the new Plugin architecture.
My objective was not to prove that the current GPT is “perfect”.
It was to create a sufficiently demanding reference point that I can reuse after migration to identify regressions, behavioral differences, improvements, or unexpected changes.
The approach
I created a second, more advanced non-regression assessment based on 100 scenarios, designed to evaluate the current Custom GPT before migration.
The complete process ultimately produced a 100-page report.
The first part of the work focused on designing and structuring the test protocol and questionnaire.
ChatGPT Work was used for the multi-step workflow and document production, with Codex involved in the technical generation of the Word documents.
The 100 responses were then evaluated in a separate conversation, deliberately separating the evaluation stage from the original response-generation process as an initial measure to reduce evaluation contamination.
The first evaluation produced an unexpected result:
300/300
A perfect score looked suspicious enough to justify another review rather than simply accepting it.
So I requested a second examination of the complete evaluation.
That review identified one constraint that had not been fully respected in Q063.
The corrected baseline became:
299/300 — 99.67%
- 99 responses: 3/3
- 1 response: 2/3
- 0 critical failures
For me, the interesting part is not really the 299/300 score.
It is the fact that the evaluation process itself demonstrated why an apparently perfect AI-generated evaluation should sometimes be challenged and independently reviewed.
The second pass actually found something the first evaluation had missed.
Why establish the baseline before migration?
A migrated Plugin may appear to work correctly while still introducing subtle behavioral changes.
Without a pre-migration reference, it becomes difficult to distinguish between:
- behavior that was already present in the Custom GPT;
- a genuine regression introduced during migration;
- an improvement;
- differences in instruction interpretation;
- differences caused by tools, retrieval, Plugin selection or execution;
- normal variability between model runs.
My current approach is therefore straightforward:
Phase 1 — Custom GPT: establish and document the baseline.
Phase 2 — Plugin: migrate the system.
Phase 3 — Non-regression: run the same benchmark again, keeping the original scenarios and scoring criteria as stable as possible.
Phase 4 — Delta analysis: investigate every meaningful difference between the two implementations instead of looking only at the final aggregate score.
That final comparison is ultimately more important to me than the current 299/300 result.
Important limitation
This benchmark should not be interpreted as an external certification.
It evaluates the written behavior tested by this particular protocol.
It does not prove that the future Plugin migration will be regression-free, and it does not yet measure every aspect of real-world Plugin execution.
The Plugin version will introduce additional dimensions that deserve their own testing, including tool invocation, capability selection, access permissions, failure handling, reference-material behavior and potentially different execution paths.
That is precisely why I wanted a documented baseline before changing the architecture.
Full context and original test publication
I documented the complete pre-migration experiment in this LinkedIn post:
Original LinkedIn post — Advanced pre-migration test for GM INFO - IA Technologies ©
The LinkedIn publication also provides access to the complete 100-page evaluation report for anyone interested in examining the test methodology and results in greater detail.
A question for the Developer Community
I would be very interested in feedback from developers who are preparing similar migrations.
How would you design a rigorous pre/post-migration regression test for a complex Custom GPT moving to Plugins?
In particular, I am interested in three questions:
-
Would you keep the complete pre-migration benchmark unchanged for the first Plugin comparison, or evolve part of the test suite to account for Plugin-specific behavior?
-
How would you further reduce evaluator bias or correlated errors when AI is involved both in producing and evaluating test results?
-
Which Plugin-specific dimensions would you consider mandatory before concluding that a migration has successfully preserved the original workflow?
My intention is to repeat the evaluation after migration and compare the results scenario by scenario, including failures, improvements and behavioral differences — not simply publish another overall score.
If this methodology is useful to others preparing the same transition, I will also share the post-migration results and delta analysis in this thread.