Research Question: RQ-external-tdd-workflows-opus5

Findings — RQ-4.7: Substituting the Inner TDD Loop (opus-5)

All figures: claim-office-example-mapping × opus-5-no-thinking (native subscription route), n=5 per cell. Both external workflows are vendored snapshots at the commits named in README.md; every statement here describes those snapshots, not the tools in general.

Overview

Primary outcome is Correctness (external); the decomposition metrics follow the binding quality metric from RQ-architecture-axis-opus5 F-1.6.

Why the quality rows carry no trophy. The gating rule awards quality and cost trophies only to cells at verification_pct = 1.0. hybrid-v2.4 sits at 0.96 with only 2/5 perfect runs — that is not a rounding remainder but a real correctness shortfall in 3 of 5 runs. hybrid-v2.4 is therefore not trophy-eligible, yet it holds the best absolute values on all four decomposition metrics. Awarding a trophy to the best eligible cell would put it on the worse value of the row, which the convention excludes — so these rows stay trophy-free. For cost, Code Mass (APP) and Smell Total the conflict does not arise: there the best cell is eligible anyway.

refactorings_applied carries no trophy: the three values are the cells' design characteristic, not a result. 33 / 4.2 / 0 is the axis itself.

F-4.7.1 — The inner loop is substitutable without losing correctness

Both external workflows reach verification_pct 1.00 in 5/5 runs. The internal baseline hybrid-v2.4 sits below that at 0.96 (2/5 perfect).

Reading: the value EXACT Coding adds lies in the example mapping and the test list, not in its own red/green plumbing. An externally authored loop carries the method on this kata at least as reliably. H5 confirmed for all three cells.

The three hybrid-v2.4 runs at 0.9333 fail on the same acceptance-scenario pattern as the case documented in RQ-4.5 F-1.4; the effect is therefore not new, but it hits hybrid-v2.4 more often than the two external cells.

F-4.7.2 — Refactor position determines decomposition monotonically

Across the three cells all four decomposition and complexity metrics fall monotonically with refactor frequency.

The clean test of position is superpowers-2026-09-04 ↔ pocock-2026-09-04 — same architecture (one inline skill), only the position varies. avg_loc_per_function 7.90 against 10.41 is the most robust single result of this pair (gap 2.51 at σ ≈ 1.3). longest_function, cognitive_max and mccabe_max point the same way but sit within 1 σ.

H1 confirmed: a refactor stage has an effect. H2 confirmed: pocock-2026-09-04 is the field's quality minimum on all four metrics. The design doctrine in the prompt ("deep modules", "small interfaces") does not replace the stage.

F-4.7.3 — The isolated subagent architecture buys decomposition, and it is expensive

hybrid-v2.4 ↔ superpowers-2026-09-04 varies architecture and mechanism at constant position (both refactor per cycle). The difference is large in both directions.

The answer to the RQ's guiding question is therefore twofold: the subagent apparatus does buy measurably better decomposition (4.49 against 7.90, ≈ 3 σ) — but at the price of 7× wallclock and 10× tokens, with worse Correctness (external) at the same time. H3 in the variant "per-cycle refactoring alone suffices" is refuted: superpowers-2026-09-04 refactors per cycle and still lands clearly outside the hybrid-v2.4 level. H4 confirmed (superpowers-2026-09-04 markedly cheaper than hybrid-v2.4).

The σ values are part of the result: relative to its mean, hybrid-v2.4 spreads three to four times wider on duration (± 1523 s) and tokens (± 42 M) than superpowers-2026-09-04. The apparatus is not only more expensive, its cost is also less predictable.

F-4.7.4 — The review stage costs more than the missing refactor saves

pocock-2026-09-04 has no refactor stage, but two review subagents at the end. It is still not the cheapest cell.

H8 (pocock-2026-09-04 is the cheapest cell) is not confirmed: pocock-2026-09-04 sits above superpowers-2026-09-04 on both wallclock and tokens, both gaps within 1 σ. Reading: the two review subagents cost about as much as the saved per-cycle refactoring yields — and deliver no code, because by construction they only report.

The token row is the one this RQ had to re-measure: pocock-2026-09-04 spends its review budget inside two subagents, and until 2026-09-30 subagent tokens were absent from total_tokens entirely (RQ-old-vs-new-exact-line-opus55 F-4.12.5). On the uncorrected figure this cell read 11.8 M and undercut superpowers-2026-09-04, which is the one arrangement of the numbers that would have argued against this finding. Dropping the review stage would give the cheapest variant of the field; with it, pocock-2026-09-04 is cost-neutral against superpowers-2026-09-04 and worse on every quality metric.

F-4.7.5 — Superpowers keeps cycle discipline but does not write tests in one block

The manual n=1 observation that Superpowers writes all tests at once does not reproduce.

test_cases_first_block is exactly 1 in 15/15 runs: no workflow starts with a test block. H6 is therefore refuted — the falsifier (test_blocks = 1 or first block ≈ total count) occurs in no run.

The step size nevertheless differs clearly: hybrid-v2.4 stays at one case per block, superpowers-2026-09-04 writes 3.36 on average. The skill's script demands "one behavior" per test; superpowers-2026-09-04 reads that noticeably more broadly than the phase commands, without skipping the red phase.

red_unverified stays low but not at zero: hybrid-v2.4 0.00, pocock-2026-09-04 0.20 ± 0.45, superpowers-2026-09-04 1.00 ± 1.73. The spread at superpowers-2026-09-04 comes from a single run; H7 holds for hybrid-v2.4 and pocock-2026-09-04 and is only weakly supported for superpowers-2026-09-04.

F-4.7.6 — Little code is not a quality signal here

The Code Mass ordering runs opposite to the complexity ordering.

pocock-2026-09-04 writes the least and packs it into the longest, most complex functions; hybrid-v2.4 writes the most and distributes it most finely. Consequence for the analysis: on this kata Code Mass (APP) alone does not separate "parsimonious" from "compacted". Without the decomposition metrics next to it, pocock-2026-09-04 would read as the leaner result — it is the unstructured one.