Official support for TDD in Codex?

Codex
,
1

I’ve been developing a workflow for using Codex on larger software projects that I think could benefit from some explicit first-class support.

The basic idea is to separate specification, testing and implementation as much as possible, rather than asking the model to implement a feature and write its own tests in the same step.

My current workflow looks roughly like this:

  1. I write the initial specification myself.

  2. I work through the specification conversationally with ChatGPT, looking for ambiguities, contradictions, underspecified behaviour and missing edge cases. The specification remains the authority here rather than an implementation.

  3. I give the specification and existing codebase to Codex and ask it to produce a comprehensive test plan. This includes normal cases, edge cases, expected failures, false-success and false-error cases, resource exhaustion, lifecycle issues, interactions between subsystems, etc.

  4. Codex builds the basic test infrastructure and implements a small initial set of tests.

  5. I run those tests against stubs or the incomplete implementation and make sure they actually fail. This is important: I’ve already caught tests which accidentally passed because a stub returning false happened to produce the expected result.

  6. The remaining tests are implemented before the corresponding functionality exists, leaving the project deliberately in a known RED state.

  7. Implementation then proceeds incrementally. Either I implement something manually, or I ask Codex to make the minimum change necessary to make one specific test pass without breaking the tests which are already green.

  8. Repeat RED → GREEN one behaviour at a time until the complete suite passes.

  9. Finally, do a separate audit comparing the specification, tests and implementation, followed by integration/stress testing.

An important rule is that once implementation begins, a failing test isn’t simply modified to accommodate the implementation. If Codex believes the test is wrong, it should identify the conflict with the specification and ask for a decision. Otherwise there’s a danger of the tests gradually adapting themselves to bugs in the implementation.

I’ve found this much more confidence-inspiring than asking an agent to “implement X and add tests”. In that workflow, the same model can make an incorrect assumption in the implementation and then encode the same incorrect assumption into its tests. You get a beautifully green test suite which proves that the implementation agrees with itself.

What I’d love to see is some official support in Codex for this kind of spec-driven, test-first incremental workflow.

For example, a project could have an explicit specification/test-plan relationship and a mode in which Codex understands that:

  • the specification is authoritative;

  • tests may intentionally be failing because functionality has not been implemented yet;

  • RED is therefore a legitimate project state rather than something that must immediately be “fixed”;

  • implementation should proceed one failing behaviour at a time;

  • passing tests should be treated as regressions if they subsequently fail;

  • tests shouldn’t be changed merely to make an implementation pass;

  • changing a test because the specification is ambiguous or contradictory should require explicit user involvement;

  • Codex can maintain a view of which specified behaviours are untested, RED, GREEN, or blocked;

  • the user can choose which behaviour to implement next;

  • Codex can be instructed to make the minimum implementation change needed for that behaviour rather than attempting the entire feature at once.

I’d also love an explicit “negative control” step for new test suites: run the tests against stubs, deliberately broken implementations, or otherwise known-wrong states and verify that the tests actually detect them. A test suite shouldn’t just demonstrate that the finished implementation passes; where practical, it should demonstrate that plausible incorrect implementations fail.

This feels particularly useful for systems programming, kernels, protocol implementations, security-sensitive code and anything else where “the tests are green” isn’t very reassuring if the implementation and tests were generated together from the same mistaken interpretation.

The ideal workflow, to me, is essentially:

SPECIFICATION
      |
      v
  TEST PLAN
      |
      v
    TESTS  -----> RED
      |
      v
IMPLEMENTATION
      |
      v
    GREEN
      |
      v
SPEC/TEST/CODE AUDIT

with the human retaining control over the transitions between those stages.

I’d be very interested in official support for working with Codex this way, rather than having to enforce the separation entirely through prompts.