Introducing Codexometer: keep track of remaining quota, session telemetry & model benchmarks!

Codex
,
22

Yes, expression challenges had some potential upside.

Did you notice that caching appears a bit volatile? Sometimes nothing is cached at all.

I am running one model-effort combination at a time repeatedly, and sometimes cache hits grow, while other times there is literally nothing cached across different combinations.

23

coming soon, multi session monitoring:

image
image1111×998 26 KB

24

That was pushed and includes additional telemetry.

Now I’ve just merged improvements to the benchmarks tab:

  • ranking on the benchmarks tab, you can rank prioritising cost or speed (but taking into account the other rank) - review the README for more info
  • simplified some options (there’s now only one collection of benchmarks)

image
image1003×910 34.6 KB

You could already order by time and cost so this allows you to blend those two in different ways.

I’ve also improved the interface to gather together the different quota views in one tab where you can select the specific style of quota view you prefer and enhancements to the status at the top to reflect quota status so it’s clear no matter what tab you are on:

image
image1009×906 17.6 KB

this quota status takes into account current known reset dates so shouldn’t flag if reset date and rate of consumption are reasonable.

25

All these resets are making my UI look boring!:slight_smile:

image
image1131×754 14.4 KB

In any case, lets get motoring people!

image
image1136×751 28.1 KB

26

I’ve added experimental quota pricing estimation.

After a while of using quota whilst codexometer is running, it will attempt to estimate your “API Equivalent” spend and also what your potential max api equivalent spend might be.

Regard this as an educated estimate and not foolproof.

The explanation of how we arrive at the figures is here: GitHub - merefield/codexometer: A terminal widget that allows you to keep track of Codex usage against your current quota · GitHub with appropriate disclaimers.

28

Looks like the 5 hour window is back for 5.3 Spark:

image
image1478×954 22.7 KB

29

Codexometer v0.9.0: expanded benchmarking and DigBench support!

The last two Codexometer releases have substantially expanded its benchmarking system—from a simple result table into a scoped benchmark runner with live, inspectable run details.

What arrived in v0.8.0

Codexometer v0.8.0 (Release Codexometer v0.8.0 · merefield/codexometer · GitHub) introduced:

  • Clickable details for completed and in-progress benchmark runs.
  • Live progress, model responses, moves, and benchmark results.
  • One-click copying of the complete run detail.
  • The ability to stop an active benchmark while retaining completed work.
  • Scope controls for selecting models and reasoning levels.

What is new in v0.9.0

Codexometer v0.9.0 (Release Codexometer v0.9.0 · merefield/codexometer · GitHub) builds on that with:

  • Separate Codexometer Core and Codexometer Extended benchmark suites.
  • Per-task selection alongside model and reasoning-level scope.
  • Clearer Run Scope and Run All workflows.
  • More complete benchmark transcripts showing prompts, tool exchanges, moves, states, final responses, token usage, and estimated API-equivalent cost.
  • Optional API-key authentication for benchmark turns.
  • Stronger timeout, interruption, redaction, and credential-isolation safeguards.

The headline addition is optional integration with DigBench (https://digbench.ai/).

DigBench—“Discovery in Games”—is a scientific-discovery benchmark containing 70 interactive games with undisclosed rules. Humans and AI agents receive the same states, available actions, and step budgets, and must discover each game’s mechanics through experimentation.

When a DIGBENCH_API_TOKEN is supplied, Codexometer discovers the available games at launch and exposes a dedicated DigBench suite. You can select particular P-x games, models, and reasoning levels, then follow the agent’s progress through each move and resulting state.

Without a DigBench token, the integration remains hidden and the regular Codexometer experience is unchanged. To start it with a token run:

DIGBENCH_API_TOKEN="<REDACTED_TOKEN>" ./codexometer

You can get an API token from their website!

Benchmark transcript capture remains restricted to benchmark-created runs. Credentials, runtime identifiers, temporary paths, terminal controls, and Codex reasoning are excluded or redacted.

DigBench is MUCH tougher than the existing benchmarks.

On one of my runs I managed to solve P-1 with 5.6 Sol high, but smaller models may struggle …

Upgrade

  go install github.com/merefield/codexometer@latest
  codexometer --version

The current release should report v0.9.0.

Credit

Thanks to @EricGT for introducing me to the new LLM-accessible game benchmark!