V2: Routing Benchmark Lab
Following my first post about my example-trained Intent Router, I have now added a new Routing Benchmark Lab / Model Arena.
Youtube Showcase:
The original router focused on:
Request → Intent → Required capabilities → Model selection
V2 adds another question:
Which model has actually performed best on this type of task?
Model Arena
The Model Arena can run the same benchmark against multiple compatible models, including models such as:
- OpenAI gpt-oss:20b
- OpenAI gpt-oss:120b
- Qwen3-Coder
- Qwen3-Coder-Next
- DeepSeek-R1
The models run sequentially so that latency measurements are not distorted by several large models competing for the same hardware.
Each benchmark stores the task, required capabilities, expected output, reasoning difficulty, and evaluation method.
Better Benchmark Marking
Instead of only checking whether an answer contains certain keywords, benchmarks can now combine several evaluation methods:
- Required or forbidden content
- JSON / structured output checks
- Exact or numeric answers
- JavaScript syntax checking
- Output constraints
- Blind manual marking
- Blind model-based marking
Each evaluation method has its own weight, and the program combines them into a final score.
Blind Model Judge
One of the main additions is an optional AI judge.
I can select any suitable reasoning/chat model as the judge. In my current setup I use gpt-oss:20b.
The judge does not know which model produced an answer.
Instead of seeing:
gpt-oss:120b produced this answer
it sees something like:
Candidate-K7P4
The judge receives the original task, the evaluation criteria, and the anonymous answer.
It then returns scores for:
- Overall quality
- Correctness
- Task completion
- Instruction following
- Relevance
- Confidence
For example:
{
"score": 94,
"correctness": 97,
"completion": 93,
"instruction_following": 95,
"relevance": 91,
"confidence": 86
}
The 94% score becomes 0.94 and is combined with the other benchmark checks.
The AI judge is treated as one source of evidence, not ground truth. If deterministic tests strongly disagree with the judge, the Arena can flag an evaluation disagreement.
Repeated Trials and Measured Profiles
Benchmarks can also be repeated several times per model.
This lets the Arena measure:
- Average quality
- Worst result
- Score variation
- Success rate
- Median latency
- Number of samples
This creates measured model profiles instead of relying only on manually entered quality and speed values.
An Interesting Early Result
In one simple p5.js coding benchmark, several models achieved almost identical quality, but their response times were very different.
For example, a smaller coding model completed the task much faster than gpt-oss:120b while achieving nearly the same score.
This is exactly what I want the router to discover.
The goal is not simply:
Choose the biggest model for difficult-looking tasks.
It is:
Choose the model that gives the best measured result for this particular type of request.
Arena-Informed Routing
The Arena now compares:
Current registry ranking
with:
Arena-informed ranking
The normal router still considers manually configured quality, speed, priority, capabilities, and endpoint health.
The Arena-informed version can also consider:
- Observed benchmark quality
- Latency
- Consistency
- Number of samples
For now, this is only shown as a preview rather than automatically replacing the live routing decision.
Current Architecture
User request
↓
Example-trained intent recognition
↓
Structured RoutingFrame
↓
Compatible models
↓
Model Arena evidence
↓
Quality + latency + consistency
↓
Arena-informed model recommendation
The project is still experimental, but V2 moves the router from choosing models mostly from declared properties toward choosing them from measured outcomes.
A possible next step is to let accumulated benchmark results directly train the model-selection policy instead of relying mainly on fixed ranking weights.