Research Question: RQ-architecture-axis-opus5

Findings — RQ-architecture-axis-opus5

Does the TDD architecture axis (inline-tdd-v1 structureless / single-context-v2 single context / hybrid-v2 hybrid / hybrid-v6 current generation) still rank the same way on opus-5 as it does on opus-4-7 — and does the decomposition metric change the answer?

Data base: 93 runs, 16 cells at n=5–10, all exit_reason: ok, completed_within_budget 100 %. The 50 opus-5 and hybrid-v6/opus-4-7 runs were produced for this RQ; the remaining opus-4-7 cells are pre-existing runs (partly Portkey-routed, see README caveat 1).

Overview

claim-office-example-mapping (correctness kata)

game-of-life-example-mapping (code-quality kata)

Caveats for reading the tables:

• Correctness gating: on claim-office, trophies for quality/efficiency metrics go only to cells at verification_pct 1.00. single-context-v2/o5 (0.79), hybrid-v6/o5 (0.95) and hybrid-v2/o5 (0.99) are exempt from the gate for the quality columns because their shortfall traces to a single acceptance scenario (F-1.4), not to stub implementations — but they carry no correctness trophy.
• cycle_count, refactorings_applied and predictions_correct_rate are n/a for inline-tdd-v1, not zero. inline-tdd-v1 prescribes no phase markers, so the parser has nothing to count. See MARKERS.md.
• Code Mass (APP) carries no trophy. It ranks the cells opposite to decomposition (F-1.6) and has no notion of nesting — established in RQ-architecture-axis-sol-pi F-1.9.
• cycle_count is ambivalent and carries no trophy.
• cost_usd is not reported: these runs went through the native Anthropic API on a subscription plan, so no per-run list price applies.

F-1.1 — H1 confirmed: the architecture ranking holds on opus-5, on both katas

The inline-tdd-v1 → single-context-v2 → hybrid-v2 → hybrid-v6 ordering on avg_loc_per_function is monotone on claim-office and near-monotone on game-of-life, exactly as on opus-4-7.

Rationale: the ranking transfers across the model generation. This is the opposite of the Sol result (RQ-architecture-axis-sol-pi F-1.6), where structureless inline-tdd-v1 beat every architecture on game-of-life and hybrid-v2 collapsed to 1.6 functions per implementation. The mechanism proposed there — the isolated refactor subagent not performing its extraction — does not apply to opus-5: hybrid-v6 averages 8.6 refactorings on game-of-life and 43.4 on claim-office.

The single-context-v2/hybrid-v2 swap on game-of-life (4.12 vs 4.54) sits inside one σ (1.00 / 0.92) and is read as "indistinguishable", not as a rank change.

Consequence: the reduction chain built on hybrid-v1 rests on an architecture opus-5 supports. H2 (Sol pattern) and H3 (metric split) are both refuted for this model.

F-1.2 — The model generation moves quality as much as the whole architecture does

Structureless TDD on opus-5 reaches complexity levels that on opus-4-7 required the full architecture.

Rationale: on both katas, inline-tdd-v1/opus-5 lands at roughly the cognitive_max of hybrid-v2/opus-4-7 (5.4 vs 5.71; 7.6 vs 6.5) — a ~15-line prompt with no agents, no skills and no phase structure now produces what previously took the full hybrid architecture. Smell Total drops to zero in all ten opus-5 inline-tdd-v1 runs, against 16.8 and 6.0 on opus-4-7.

The architecture still adds on top of that (F-1.1), but the baseline it starts from has moved by a comparable margin. Workflow recommendations calibrated on opus-4-7 absolute values do not carry over.

F-1.3 — hybrid-v6 wins the quality axis on both katas and costs 6–36× the baseline

hybrid-v6 takes the trophy on every quality metric except longest_function on claim-office. The cost side is where the decision actually sits.

Rationale: the quality gain is real and consistent, but its size (0.16–0.56×) is an order of magnitude smaller than the cost (6–36×). On game-of-life, where correctness is tied at 1.00 across all cells, the trade is 9.5× the tokens for a 1.9× better decomposition. On claim-office it is 36× the tokens for 2.9× — and 0.95 instead of 1.00 external correctness.

The intermediate cells are the interesting ones for practice: hybrid-v2 reaches avg_loc_per_function 4.04 on claim-office at 94 M tokens — 58 % of hybrid-v6's cost for 80 % of its decomposition gain over inline-tdd-v1.

Token figures here count subagent consumption; the three structured arms all spawn subagents and were recorded short until 2026-09-30 (RQ-old-vs-new-exact-line-opus55 F-4.12.5). The correction widens every factor in this finding rather than narrowing it, because the baseline arm spawns none.

F-1.4 — One acceptance scenario fails across workflows and accounts for most of the correctness spread

Seven opus-5 runs on claim-office pass all internal tests but miss external scenarios. Six of the seven fail on the same one.

Rationale: 14-family-steinheim fails for subagents-v1, single-context-v2, hybrid-v2 and hybrid-v6 alike on opus-5 — it is a kata/model interaction, not a workflow defect. Any correctness comparison between opus-5 cells on claim-office therefore measures partly this one scenario. The hybrid-v6 figure of 0.95 means "14 of 15 scenarios in 4 runs, 15 of 15 in one", not "systematically less correct".

The exception is the one single-context-v2 run at verification_pct 0: it stopped after 2 cycles with 6 functions and 60 passing self-written tests. That is the internal/external divergence documented as F-1.2 in RQ-architecture-axis-sol-pi, and it is what drags the single-context-v2 cell mean to 0.79 (σ 0.39) — the cell is bimodal, not uniformly mediocre.

F-1.5 — single-context-v2 is the least reliable architecture on opus-5, hybrid-v2 the most efficient compromise

Rationale: single-context-v2's shared single context has a failure mode where the model declares the task complete after 2 cycles. hybrid-v2 and hybrid-v6, which isolate the refactor phase, do not show it in 10 runs. This is the same instability the Sol RQ documented for single-context-v2 (F-1.4 there, refactorings_applied σ 17.4) and it reproduces on a different model and harness.

Against that, hybrid-v2 delivers 94 M tokens and 44 minutes for avg_loc_per_function 4.04 — against hybrid-v6's 162 M and 93 minutes for 3.21. The marginal decomposition gain from hybrid-v2 to hybrid-v6 (0.83 LoC per function) costs 72 % more tokens and 111 % more wallclock.

F-1.6 — Code Mass (APP) ranks the cells opposite to decomposition, on this model too

Rationale: the cell with the best decomposition carries the highest Code Mass on both katas. APP measures compactness and has no notion of nesting or naming; a workflow that extracts more named functions scores worse on it by construction. This replicates RQ-architecture-axis-sol-pi F-1.9 on a different model and harness, which strengthens the methodological conclusion: Code Mass (APP) must not be read as a quality metric in architecture comparisons.

F-1.7 — opus-5 writes more code than opus-4-7 under the same workflow

Rationale: in five of six matched cells opus-5 produces more code mass than opus-4-7 for the same workflow and kata, most strongly under hybrid-v6. The additional volume comes with better decomposition, not worse — but it means the "same work, better organised" reading of the architecture gain is only partly right. Part of what hybrid-v6 buys on opus-5 is more code, distributed across more functions.

Caveat: the opus-4-7 single-context-v2 and hybrid-v2 cells are pre-existing Portkey-routed runs; routing is assumed not to affect code volume, but it was not verified.