Research Question: RQ-test-list-dimensions-opus-native

RQ-test-list-dimensions-opus-native — Findings

Overview

Both cells use claim-office-example-mapping, native opus-5-no-thinking, and the same single-context PTDD workflow. The only treatment is the v1.6 independent-dimensions cross-check before the test list is declared complete. Higher is better for Correctness (external) and prediction accuracy; lower is better for decomposition and efficiency metrics.

Each cell pools two batches of five runs on Claude Code 2.1.267: this RQ's own batch (v1.5 on 2026-09-15, v1.6 on the morning of 2026-09-16) and the replication batch of RQ-test-list-dimensions-replication, which runs the same workflow × model × kata cells (both arms on the afternoon of 2026-09-16).

The correctness trophy reflects the shallower lower tail: v1.6 has one run at 0.933 and nine perfect runs, whereas v1.5 has three imperfect runs down to 0.733. The efficiency means favour v1.6, but every difference lies within the v1.5 standard deviation and is carried by the first v1.5 batch's outliers, so no efficiency trophy is awarded. No trophy is awarded for test volume, process counts, Code Mass (APP), or code size because their direction is ambiguous. Static-quality differences remain within the observed spread and therefore receive no trophy. The reported cycle_count is omitted because the single-command parser path produced values incompatible with the refactor markers in several runs; it is not a valid treatment measure here.

F-4.10.1 — The dimensions cross-check removes the catalogue-omission failure but not every completeness miss

All internal suites remain green in both cells and every run completes within budget. v1.6 reaches full Correctness (external) in nine of ten runs; v1.5 in seven of ten.

The imperfect runs fall into two failure modes. The 0.733 v1.5 run omitted Staff and Potion from the insurance-value catalogue while testing their premiums, and its CLI exits with an error on three scenarios. No v1.6 implementation reproduces that omission. The other three misses — two v1.5 runs and one v1.6 run, all from the pooled afternoon batch — each fail only scenario 14-family-steinheim with a wrong output rather than a crash. The cross-check therefore removes the catalogue omission but does not address the Steinheim reading, which appears in both arms at a similar rate.

The mechanism is not perfectly compliant in every transcript. One v1.6 run explicitly notes that Staff and Potion insurance values lack direct discriminating tests because no listed cap example exercises them, despite the new command treating independently specified catalogue entries as candidates for separate coverage. Its implementation nevertheless includes both values and passes externally. The outcome supports v1.6 as the safer current Opus PTDD variant on Claim Office, but it does not establish that explicit matrix tests are the sole causal mechanism, and it does not make v1.6 complete in every run.

F-4.10.2 — Completeness hardening costs no efficiency and narrows the run-to-run spread

The added coverage review does not impose an observable efficiency penalty. v1.6 is faster on average, uses fewer tokens and costs less, but the mean differences lie within the v1.5 standard deviation.

The shift is larger in stability than in the mean. v1.6 stays between 11.1 M and 34.4 M tokens and between $9.39 and $22.71; v1.5 ranges from 10.9 M to 96.2 M and from $9.46 to $65.09. The v1.5 tail comes from its first batch: its pooled second batch lies between 13.3 M and 21.5 M tokens and $10.95 and $16.80, inside the v1.6 range. A prompt addition cannot mechanically guarantee lower consumption, so the reduction should be read as observed behaviour rather than an intrinsic cost law. It is nevertheless enough to reject the hypothesis that the cross-check necessarily spends away PTDD's price/performance advantage.

F-4.10.3 — Product shape remains within the prior workflow's spread

The dimensions cross-check changes test planning but produces no resolved static-quality regression. Typical functions are slightly longer in v1.6 and the longest_function is slightly higher, while Cognitive Complexity and mccabe_maxs are tied; all differences overlap the run-level spread.

The larger v1.6 Code Mass (APP) has broad overlap with its own variance and is not a quality verdict. Production LoC differs by less than one standard deviation, and both cells remain smell-free with low complexity peaks. The treatment therefore acts primarily on completeness and execution behaviour rather than imposing a new implementation architecture.

F-4.10.4 — The cross-check expands test expression without creating a mechanical cross product

v1.6 produces more executable tests and substantially more Test LoC, but test volume varies widely and is not itself the success criterion.

The increase is consistent with a more explicit coverage review, while the absence of explosive growth is consistent with the instruction to use representative cases where dimensions cannot fail independently. The direct Staff/Potion coverage caveat shows that test count alone cannot demonstrate treatment fidelity. The binding outcome remains external correctness — nine of ten v1.6 runs perfect and none below 0.933 — not the amount of test code produced.