Research Question: RQ-test-list-dimensions-replication

Findings — RQ-test-list-dimensions-replication

Overview

Cell means ± standard deviation at n=10 per cell. v1.5 is the PTDD workflow without the independent-dimensions cross-check in the test-list phase; v1.6 adds it and changes nothing else.

The two platforms are separate tables and are never compared with each other. Absolute duration, token and cost levels differ by harness and route, so only the within-platform contrast is causal. Trophies are awarded within a table.

Complexity, unit size, duration, cost and missed mutants are lower = better; Correctness, Mutation Score and prediction accuracy are higher = better. Production LoC, Code Mass (APP), function count, test count, Test LoC, mutants_total and the refactoring count have no unambiguous direction and receive no trophy; they appear as context.

Correctness guard — passed. All four cells clear the 0.90 gate. Correctness (internal) and completion within budget are 100 % in all four.

mutants_survived is split into its two parts. A Survived mutant means a test ran the mutated line and still passed — a weak assertion. An uncovered one means no test reaches the line at all — untested code.

Native Opus 5 / Claude Code

GPT-5.6 SOL / pi

Rows where the two columns are identical (cognitive_max and mccabe_max on Opus, Smell Total on both) carry no trophy: there is no contest to win.

The SOL v1.6 Mutation Score is reported at n=9. One run in that cell scored 0.00 with 112 of 112 mutants surviving against a green suite at perfect external correctness — an instrument failure rather than a measurement of the suite, and excluded here. summary.md carries the unfiltered n=10 figure of 0.78 ± 0.28, which the σ alone marks as describing two populations.

F-1.6.1 — The platform interaction replicates, and it lives in the correctness floor

The effect that motivated this RQ survives n=10 per cell. It is not a shift in the mean but the removal of a bad tail.

On Opus the cross-check lifts the worst run from 0.73 to 0.93 and empties the sub-0.90 band, cutting the standard deviation by a factor of four. The v1.5 tail is one run at 0.73 plus two at 0.93; v1.6 has a single run at 0.93 and nine perfect ones. The mean moves only 0.03, so a reader of means alone would conclude the treatment does nothing.

On SOL there is nothing to lift: both cells sit at a perfect score with zero variance across twenty runs. The interaction is therefore not a difference in how the two platforms respond to the cross-check — it is that only one of them had a completeness problem for the cross-check to solve.

This was small-sample variance in neither direction. At n=10 the effect is the same one the earlier sample showed, and it is a variance effect rather than a level effect.

F-1.6.2 — The cross-check also halves cost variance on Opus

The same tail-trimming shows up in the price.

The v1.5 cell contains a run costing $65.09 — nearly three times its own cell mean — and its σ of 17.33 is larger than v1.6's entire mean. The cross-check removes that mode: the worst v1.6 run costs $22.71, and the cell is 30 % cheaper on average with a quarter of the spread.

A completeness step that makes the test list explicit up front therefore pays for itself here rather than costing extra, which is the opposite of what an added phase usually does. On SOL cost is unchanged ($4.78 against $4.76) — the same pattern as F-1.6.1: no bad mode to remove.

F-1.6.3 — Product structure is untouched on Opus and slightly worse on SOL

The cross-check acts on the test list, not on the production code, and the structural metrics confirm it.

On Opus cognitive_max and mccabe_max are identical to the second decimal across twenty runs, and the size measures differ by less than either cell's spread. Structure is genuinely unaffected.

On SOL the two size measures move against v1.6 by more than the better cell's standard deviation — avg_loc_per_function 5.72 → 6.42 (σ 0.52) and median_loc_per_function 4.35 → 5.30 (σ 0.58). Combined with F-1.6.1, this makes v1.6 a cell with no correctness benefit and a mild structural cost on that platform.

F-1.6.4 — More tests, weaker suites: the cross-check lowers test strength on both platforms

This is the finding that does not follow from the RQ's original question, and it points the other way from every other result here.

v1.6 writes 18 % more tests on Opus and 17 % more on SOL, and its suites catch less. On Opus the absolute number of missed mutants more than doubles, from 4.5 to 10.4 per run, against a mutant population that grows only 6 %. Both components rise, and the uncovered part rises fastest: 1.9 → 5.8, a tripling of mutants no test reaches at all.

The pair matters here. On Opus the score gap of 0.04 corresponds to 5.9 more missed changes per run — the ratio understates the change because v1.6's denominator is larger. On SOL the score gap of 0.03 corresponds to 5.0 more missed changes, with a 14 % larger population.

A plausible mechanism, not established by this RQ: the cross-check asks for test cases along independent dimensions, which produces more but flatter cases — broad enumeration of a dimension rather than deep interrogation of a behaviour. The test count rises while assertion depth falls. What is established is the direction and that it holds on both platforms, which is more than any other outcome here does.

This qualifies the recommendation from F-1.6.1. The cross-check buys a correctness floor on Opus and costs test strength on both platforms. Where the floor is already perfect — SOL — it is a net loss: no correctness gain, slightly worse structure, measurably weaker suites.

Caveats

• Single kata and prompt. All conclusions are about Claim Office under Example Mapping.
• cost_usd is a list-price comparison value, not an invoice. For pi runs it comes from token counts and the price table, never from pi's own inline figures.
• predictions_correct_rate is a pooled rate, not a per-run mean, so it carries no standard deviation and its trophies rest on a smaller margin than the other rows.
• Mutation Score was measured after the fact, in a single pass over all forty runs, and was not part of the RQ's original outcome set. It is the source of F-1.6.4 and of nothing else here.
• The SOL v1.6 mutation cell is n=9. The excluded run is an instrument failure, documented in the Overview.