Codexometer: track your quota, usage, orchestrate sessions & benchmark models!

Yes, expression challenges had some potential upside.

Did you notice that caching appears a bit volatile? Sometimes nothing is cached at all.

I am running one model-effort combination at a time repeatedly, and sometimes cache hits grow, while other times there is literally nothing cached across different combinations.

coming soon, multi session monitoring:

That was pushed and includes additional telemetry.

Now I’ve just merged improvements to the benchmarks tab:

  • ranking on the benchmarks tab, you can rank prioritising cost or speed (but taking into account the other rank) - review the README for more info
  • simplified some options (there’s now only one collection of benchmarks)

You could already order by time and cost so this allows you to blend those two in different ways.

I’ve also improved the interface to gather together the different quota views in one tab where you can select the specific style of quota view you prefer and enhancements to the status at the top to reflect quota status so it’s clear no matter what tab you are on:

this quota status takes into account current known reset dates so shouldn’t flag if reset date and rate of consumption are reasonable.

All these resets are making my UI look boring!:slight_smile:

In any case, lets get motoring people!

I’ve added experimental quota pricing estimation.

After a while of using quota whilst codexometer is running, it will attempt to estimate your “API Equivalent” spend and also what your potential max api equivalent spend might be.

Regard this as an educated estimate and not foolproof.

The explanation of how we arrive at the figures is here: GitHub - merefield/codexometer: A terminal widget that allows you to keep track of Codex usage against your current quota · GitHub with appropriate disclaimers.

Looks like the 5 hour window is back for 5.3 Spark:

Codexometer v0.9.0: expanded benchmarking and DigBench support!

The last two Codexometer releases have substantially expanded its benchmarking system—from a simple result table into a scoped benchmark runner with live, inspectable run details.

What arrived in v0.8.0

Codexometer v0.8.0 (Release Codexometer v0.8.0 · merefield/codexometer · GitHub) introduced:

  • Clickable details for completed and in-progress benchmark runs.
  • Live progress, model responses, moves, and benchmark results.
  • One-click copying of the complete run detail.
  • The ability to stop an active benchmark while retaining completed work.
  • Scope controls for selecting models and reasoning levels.

What is new in v0.9.0

Codexometer v0.9.0 (Release Codexometer v0.9.0 · merefield/codexometer · GitHub) builds on that with:

  • Separate Codexometer Core and Codexometer Extended benchmark suites.
  • Per-task selection alongside model and reasoning-level scope.
  • Clearer Run Scope and Run All workflows.
  • More complete benchmark transcripts showing prompts, tool exchanges, moves, states, final responses, token usage, and estimated API-equivalent cost.
  • Optional API-key authentication for benchmark turns.
  • Stronger timeout, interruption, redaction, and credential-isolation safeguards.

The headline addition is optional integration with DigBench (https://digbench.ai/).

DigBench—“Discovery in Games”—is a scientific-discovery benchmark containing 70 interactive games with undisclosed rules. Humans and AI agents receive the same states, available actions, and step budgets, and must discover each game’s mechanics through experimentation.

When a DIGBENCH_API_TOKEN is supplied, Codexometer discovers the available games at launch and exposes a dedicated DigBench suite. You can select particular P-x games, models, and reasoning levels, then follow the agent’s progress through each move and resulting state.

Without a DigBench token, the integration remains hidden and the regular Codexometer experience is unchanged. To start it with a token run:

DIGBENCH_API_TOKEN="<REDACTED_TOKEN>" ./codexometer

You can get an API token from their website!

Benchmark transcript capture remains restricted to benchmark-created runs. Credentials, runtime identifiers, temporary paths, terminal controls, and Codex reasoning are excluded or redacted.

DigBench is MUCH tougher than the existing benchmarks.

On one of my runs I managed to solve P-1 with 5.6 Sol high, but smaller models may struggle …

Upgrade

  go install github.com/merefield/codexometer@latest
  codexometer --version

The current release should report v0.9.0.

Credit

Thanks to @EricGT for introducing me to the new LLM-accessible game benchmark!

Please note, Codexometer is now distributed as a proper compiled binary release now, so install and upgrade command has changed, see the OP for details: Introducing Codexometer: keep track of remaining quota, session telemetry & model benchmarks!

This makes installation much faster and more convenient.

Please note there is currently an issue whereby if you are running a benchmark that is consuming Codex credits (see README) the API EQ estimates on the Quota views do not presently take the token usage into account.

The effect of this is to underestimate the API EQ cost of the API credits consumed as well what 100% would represent.

The quota consumption % remains correct, however.

With the new, very hard DigBench benchmarks, this could underestimate $ spend by a factor of 8!

I’m working on a PR to address this.

The current workaround is to ignore those estimates if you’ve been running benchmarks in the current session.

If you close and reopen Codexometer, and do not run any benchmarks, the number should be much more accurate (and a lot higher)

:+1:

and to celebrate this:

and

I’ve added a working reset button to Codexometer :sparkles:

This normally doesn’t show until you’ve used 80%, but there’s a CLI switch to reduce that threshold - if you must! :wink:

NB you have to click the confirmation after your first click :sweat_smile:

Could this be the final Astra-launch-week feature for Codexometer??

Localisation support for some popular languages:

UK English, Dutch, German, French, Italian, Spanish, Russian, Japanese and Simplified Chinese.

Will accept PRs for additional languages and let me know of any egregious errors :sweat_smile:

Codexometer v0.14.0: your retro Codex command centre :joystick:

Codexometer started as a little quota dashboard to keep beside your Codex terminal. Now it’s getting a captain’s chair :seat: :saluting_face:

The new Monitor lets you see what your sessions are doing - and, with the shared local app-server setup, respond without hopping between terminals:

  • :eyes: Read the latest activity, replies and questions per session.
  • :vertical_traffic_light: Spot TURN COMPLETE, INPUT NEEDED and APPROVAL NEEDED.
  • :magnifying_glass_tilted_left: Inspect commands and their justifications before approving.
  • :white_check_mark: Handle supported approvals, answer questions and send follow-up instructions directly from Monitor.

Keep things compact, expand a session’s context, or open the full-detail view when it needs your attention. Multiple sessions, one little retro command centre.

Also aboard: Swedish, Norwegian Bokmål, Turkish, Estonian and Finnish - bringing us to 14 interface languages. :globe_showing_europe_africa:

Ordinary CLI sessions still work with local monitoring; interactive controls need the shared app-server setup documented in the README.

Release highlights and upgrade instructions →

If Codexometer makes your fleet easier to manage, a GitHub :star: is always appreciated!

Gave you a :star:! Nice work and appreciate the addition of Swedish and Finnish​:grinning_face_with_smiling_eyes:

But I missed the Danish! :man_facepalming:t3:. Next time.

Some attempts at QoL:

Feel free to provide feedback on this.

Three new languages and some refinement of the Monitor command interface