Research Question: RQ-current-ptdd-vs-exact-opus-native

RQ-current-ptdd-vs-exact-opus-native — Findings

Overview

Both cells use claim-office-example-mapping with native opus-5-no-thinking. Lower is better for decomposition and efficiency metrics; higher is better for Correctness (external). Code Mass (APP) is shown as context without a trophy because lower mass can reflect either parsimony or omitted structure.

The PTDD cell pools ten runs from two batches on the same native route and Claude Code 2.1.267: five from this RQ's batch (2026-09-15) and five from the RQ-test-list-dimensions-opus-native batch (2026-09-16), which ran the identical workflow, model and kata. The default cell's 18 runs span 2026-08-10 to 2026-09-10.

Correctness is statistically indistinguishable at the cell-mean level, so both cells receive the trophy. The PTDD cell remains eligible for quality and efficiency comparison because its mean Correctness (external) is above the 0.90 gate, but its single 0.733 run is treated explicitly in the findings below. No trophy is awarded where the difference is within the observed spread, every cell ties, the metric is ambivalent, or the workflows measure process activity through different execution mechanisms. Production LoC, Test LoC, Code Mass (APP), cycle count, and refactor-phase count are therefore contextual rather than winner metrics. The PTDD cycle count is additionally noisy: one run records a single cycle alongside 132 predictions, a marker-count artefact rather than a one-cycle run.

F-4.9.1 — The Opus default decomposes more strongly than the transferred PTDD workflow

The default EXACT Coding workflow produces substantially smaller functions than the current SOL PTDD workflow on native Opus 5. The separation is clear in both the mean and median function-size measures, while the longest_function remains within the broad run-level spread.

The mean-function gap of 1.72 LoC exceeds the standard deviation of either cell, and the median gap is larger still. The near-tie in longest_function means the distinction is distributional: the default consistently creates more small functions rather than merely suppressing the single largest function. The mandatory domain-boundary trial in PTDD therefore does not reproduce the default workflow's finer-grained decomposition when transferred to Opus.

F-4.9.2 — PTDD trades decomposition for lower runtime and token use

PTDD completes the same kata with about a third of the tokens and just over half the wallclock time of the Opus default. Its list-price comparison is lower on average, but much less predictable.

The one-context PTDD workflow avoids the default's repeated isolated-refactor context cost. The gain is not stable per run: PTDD ranges from 10.9 M to 96.2 M tokens and from $9.46 to $65.09, whereas the default has a narrower relative spread. PTDD is therefore cheaper in expectation in this sample, not a deterministic low-cost path.

F-4.9.3 — Equal mean correctness hides a PTDD completeness failure

The two workflows have nearly identical mean Correctness (external), and every run passes its internal test suite. Their lower tails differ: PTDD has one run at 0.733, while all 18 default runs remain at or above 0.933.

The PTDD outlier is not an infrastructure failure: it completed within budget, built the CLI, and finished with 59 passing internal tests. It omitted externally verified behavior despite an apparently complete local suite. Consequently, the mean supports no correctness winner, but the default has the stronger observed floor and lower correctness variance.

F-4.9.4 — Complexity peaks tie despite different code mass and test volume

The decomposition difference does not become a broad static-complexity advantage. Both workflows produce zero detected smells, nearly identical cognitive_maxs, and overlapping mccabe_maxs. PTDD emits less production and test code and lower Code Mass (APP), but these quantities do not establish better design on their own.

PTDD's lower Code Mass (APP), Production LoC, and Test LoC coexist with larger typical functions and one externally incomplete run. The safest interpretation is a smaller implementation and test surface, not an unqualified quality improvement. The default spends additional code and context on finer decomposition and broader test expression without reducing the already-low complexity peaks.