Research Question: RQ-model-novel

RQ-model-novel Findings

Model comparison of Fable 5 vs opus-4-8-no-thinking vs opus-4-7-no-thinking vs opus-4-6-portkey-no-thinking (each no-thinking) on claim-office-example-mapping × exact-subagents-v1-cc.

Overview

Primary outcome verification_pct (Correctness (external), higher = better); secondary code quality (cognitive_max/mccabe_max/smell_total/longest_function, lower = better) and cost (total_tokens/duration_seconds, lower = better).

verification_pct: 4-6 (0.93) and 4-8 (0.92) are indistinguishable within 0.1 σ → both 🏆; opus-5 (0.88) and fable-5 (0.83) are in the midfield, 4-7 (0.67) falls clearly behind. Correctness gating of the quality trophies: opus-5 gets the cognitive_max and smell_total trophies: with 0.88 correctness (one run 15/15, all ≥ 11/15) it delivers real, often complete implementations while achieving the lowest cognitive_max (2.8) and Smell Total (0.2) of all correct models — clearly below opus-4-8 (7.4 / 1.2). opus-4-8 keeps the mccabe_max trophy (7.0). fable-5 does have equally low raw values but deliberately receives no trophy — at 0.83 correctness and never a full implementation (max 14/15) that would be misleading; the parenthesized values mark leading numbers without full correctness coverage. opus-4-7's lower complexity would be equally misleading — its bimodal distribution (runs down to 3/15) partly depresses the complexity through incomplete implementations.

Cost trophies (not correctness-gated, but to be read with caution): The total_tokens trophy is shared by opus-4-6 (33.9 M) and 4-7 (36.1 M), a gap well inside 4-7's σ. fable-5 sits third at 44.5 M and carries the largest spread in the field (σ 19.3), so its cost is not merely mid-field but unpredictable. opus-5 is the most expensive at 69.0 M, above opus-4-8's 62.0 M. The duration_s trophy stays with 4-7 (3693 s); opus-5 is in the midfield at 5931 s, partly inflated by server-side 529 backoffs (one timeout run at 7201 s). 4-7's token value is, as before, to be read with caution (low cost with a partly aborted spec) — and it is the one cell where that caution now cuts the other way, since it is cheap and leads the column.

These figures were re-derived after the subagent-token correction (RQ-old-vs-new-exact-line-opus55 F-4.12.5). exact-subagents-v1-cc isolates all four phases, so every cell was recorded short, and not by the same factor: 2.3× for opus-4-6 against 3.3× for fable-5, because the models differ in how much they delegate. That is what moved the trophy off fable-5, which had held it at 13.4 M.

F-model-novel.1 — opus-4-8 and opus-4-6 Solve claim-office Reliably, opus-5 and fable-5 in the Midfield, opus-4-7 Does Not

On exact-subagents-v1-cc, opus-4-8 (0.92, σ 0.09) and opus-4-6 (0.93, σ 0.08) reach nearly identical, high correctness with a tight spread. opus-4-7 is clearly below at 0.67 (σ 0.36) and is bimodally distributed (4 perfect runs, 6 between 0.20–0.87). opus-5 (0.88, σ 0.11, n=5) and fable-5 (0.83, σ 0.10) are in between — above 4-7, but below the two leading models. opus-5 reaches the full implementation (15/15) in one run, the rest at 11–14/15 (one timeout run at 11/15); fable-5 reaches max 14/15 and never 15/15. All opus-4-8 and opus-4-6 runs are ≥ 0.80 with runs up to 15/15; opus-4-7 has no comparably consistent high cluster.

The naive expectation "newer model = monotonically better" does not hold: neither the middle model (4-7) nor the newest generation (opus-5, fable-5) is in front, but the two Opus 4 edges (4-6 and 4-8). On this novel kata with subagents-v1, opus-4-7 is the weakest of the Opus models — consistent with the RQ-regression observation that without enforced test-list completeness subagents-v1 can lose entire spec operations on opus-4-7 (for the mechanism see F-model-novel.4). opus-5 (0.88) does not rank at the top despite being the newest generation: it delivers the cleanest code of all correct models (see overview + the F-model-novel.6 profile) but on average does not reach the spec fidelity of 4-6/4-8 — the claim-office novel kata rewards spec completeness, not code elegance.

opus-4-8 settles the question of whether the high 4-6 correctness was merely conservative parsing: the model reaches the same correctness without paying the quality price (see F-model-novel.5). The 4-6 strength on claim-office is thereby reproducible as "spec fidelity", not as a routing or generation artifact.

Limitation: opus-4-6 runs via Portkey (n=5, of which 4 within budget — one timeout), opus-4-7 and opus-4-8 via the native API (n=10 and n=5 respectively). opus-5 runs natively (n=5, 4 within budget — one timeout due to server-side 529 backoffs, not due to model slowness). The routing difference remains a potential confound between 4-6 and the native models; between 4-7, 4-8 and opus-5 (all native) it does not exist.

F-model-novel.2 — The Workflow × Model Interaction Is the Dominating Effect

The finding "opus-4-6 ≈ opus-4-8 ≫ opus-4-7" holds only for exact-subagents-v1-cc. For opus-4-6 and opus-4-7 the picture reverses on exact-hybrid-v1-cc:

Neither "opus-4-7 is better" nor "opus-4-6 is better" is a tenable statement — the workflow determines which model comes out in front. single-context-v1 is constant independently of the model (0.87). Model comparisons without workflow control are not generalizable on novel katas.

Mechanism: exact-hybrid-v1-cc delegates orchestration to the model (skill invocation in the shared context) — opus-4-7 masters this, opus-4-6 loses the claim half of the spec in ~40 % of the runs (implements only quote, ignores the claim operation entirely, yet still tests_passing=true because the internal tests only cover quote). subagents-v1 gives every phase an explicit subagent prompt — opus-4-6 benefits from this structure.

opus-4-8 has so far only been collected on subagents-v1; whether it shares the workflow sensitivity of 4-6/4-7 or (as its consistently high subagents-v1 correctness suggests) is more robust across workflows is open — hypothesis in the README.

F-model-novel.3 — Correctness Differentiates More Strongly Than Code Quality

On game-of-life (RQ-model-quality), code quality separates the models at perfect correctness. On claim-office, correctness itself separates the models: opus-4-7's verification_pct spread (0.20–1.00) is the largest model difference of the entire RQ, while all four models have internal tests 100 % green (tests_passing 100 % for all). External correctness therefore uncovers spec gaps that the internal tests do not see — the "harder challenge" exposes differences that were invisible on GOL. fable-5 is a good example: 100 % green tests (up to 45 tests/run) at 0.83 external correctness and never 15/15 — a green internal suite ≠ a complete spec (F-model-novel.6).

F-model-novel.4 — A More Precise Mechanism on opus-4-7: Test-List Completeness, Not Subagent Isolation

Refines F-model-novel.2. From RQ-tdd-correctness on claim-office × opus-4-7-portkey-no-thinking (35 runs):

verification_pct: higher = better; 🏆 = best value. passed_immediately: lower = better (discipline gain, see observation 3); 🏆 = lowest value. The bold on subagents-v2 marks the finding hero (it closes the subagents-v1 gap), not the column maximum.

† Workflows archived in experiments/workflows/_archive/ (2026-05-22). Removed from the RQ-tdd-correctness frontmatter; the completed runs are retained as evidence for these findings.

Observation 1 — subagents-v2 closes the subagents-v1 gap almost completely. The only substantive difference between subagents-v1 → subagents-v2 is in the test-list subagent: the explicit obligation "Every requirement in the spec MUST produce at least one test … Every operation the spec names". Nothing else. With that, subagents-v2 reaches single-context-v1/hybrid-v1 level on 4.7 (0.96), with a considerably tighter spread (σ 0.09 vs. 0.36).

Observation 2 — subagents-v2.1 (additional shared example-mapping) brings no improvement over subagents-v2. subagents-v2.1 inherits the test-list fix from subagents-v2 and additionally writes example-mapping/<feature>.md, which red and green read before each phase. Nevertheless only 0.71 — within the subagents-v1 spread and clearly below subagents-v2. The bimodal distribution (σ 0.41, min 0, max 1.00) suggests that the additional spec view in red/green destabilizes rather than helps. If missing spec context in the red/green subagent were the reason for the subagents-v1 weakness, subagents-v2.1 would have to be better than subagents-v2 — it is not.

Observation 3 — subagents-v2.1 lowers passed_immediately (14.0 vs. subagents-v2's 22.2) without improving verification_pct. The spec view in red/green evidently reduces green anticipation (fewer tests that are already green before red activates them). But this discipline gain does not translate into correctness — which is an indication that passed_immediately is a secondary mechanism, not the dominant one.

Refined hypothesis (replaces "overly creative" for 4.7):

On opus-4-7 the dominant failure mode in subagents-v1 is an incomplete test list that omits an entire spec operation (e.g. claim) — analogous to the documented 4.6 failure on hybrid-v1 (claim half lost, F-model-novel.2). As soon as the test list is complete (subagents-v2), the subsequent subagent phases (red/green) are sufficiently steered by the activated tests. Additional spec context in red/green (subagents-v2.1) brings no added value and even degrades — a possible mechanism: red/green re-interpret the spec instead of orienting themselves strictly on the active test.

Consequence for F-model-novel.2: The actual mechanism lies in the test-list step. hybrid-v1 is not superior on 4.7 because red is not a subagent, but because the single-context setup structurally prevents incomplete test lists.

Resolved (2026-05-22): Two subagents-v2.1.1 runs show verification_pct 0.73 and 0.67 — both within the subagents-v2.1 range and clearly below subagents-v2. passed_immediately fell to 12 (vs. subagents-v2.1: 14, subagents-v2: 22.2), but correctness does not benefit. The wallclock rose by ~50 % compared to subagents-v2.1 (5500s vs. 4500s), because refactor takes over more work when green only fakes (40 refactorings instead of 25.7).

Final conclusion: Green anticipation is NOT a second independent failure mode on 4.7 — lowering passed_immediately via fake-it is real but does not translate into correctness. Test-list completeness (subagents-v2) is the only relevant workflow lever on 4.7 for claim-office correctness. The subagents-v2.1 branch (shared context files for red/green) was discarded on 2026-05-22; the workflow definitions are archived in experiments/workflows/_archive/. For the consequence for future workflow designs see research/workflow-dev/workflow-construction.md → "Shared context files for red/green are not a correctness lever on 4.7".

F-model-novel.5 — opus-4-8 Buys the Best Code Quality at ~2× the Token Cost

At almost the same correctness as opus-4-6 (0.92 vs 0.93), opus-4-8 delivers the best code quality of all three models: cognitive_max 7.4 (vs 4-6: 22.2, 4-7: 10.5), mccabe_max 7.0 (vs 10.6 / 7.9), smell_total 1.2 (vs 5.6 / 1.8). The cognitive_max (longest_function) is at 28.4, close to 4-7 (25.0) and clearly below 4-6 (50.8). opus-4-8 thereby unites the spec fidelity of 4-6 with code quality that reaches or exceeds 4-7 — the trade-off "conservatively correct (4-6) vs. clean but unreliable" from the two-model picture dissolves on the newest generation.

The price is on the cost axis: opus-4-8 consumes on average 62.0 M total_tokens — about 1.8× as much as 4-6 (33.9 M) and 4-7 (36.1 M) — and takes the longest at 5264 s (vs 4416 / 3693 s). At the same time it writes the most code (code_mass 820, lines_of_code 303 vs. 4-7: 626 / 194) at the lowest complexity per function — i.e. the extra volume is distributed over more, simpler functions, not over a few complex ones. For correctness- and quality-critical work on subagents-v1, opus-4-8 is the strongest choice; where token budget or latency binds, opus-4-6 remains the cheaper option at comparable correctness.

Limitation: n=5. The cognitive_max mean (7.4) is pulled up by one outlier run (18), which is at the same time the only opus-4-8 run with verification_pct 0.80; the remaining four are at 3–7. The highly complex outlier and the weakest correctness run therefore coincide.

F-model-novel.6 — fable-5: The Cleanest, Most Thoroughly Tested Code — But Never the Full Spec

On claim-office, fable-5 shows an internally consistent but paradoxical profile: it writes by far the cleanest code of all four models and at the same time the least spec-complete among the usable models.

• Quality (leading, but un-trophied): cognitive_max 4.0 and mccabe_max 4.2 are the lowest in the RQ — about one fifth of opus-4-6 (22.2 / 10.6) and clearly below opus-4-8 (7.4 / 7.0). longest_function 14.8 (vs. 4-8: 28.4, 4-6: 50.8) and smell_total 0.2 (vs. 4-8: 1.2, 4-6: 5.6) are also best values. The code is at the same time thoroughly tested (all runs tests_passing=true, up to 45 tests) on the most compact code base (lines_of_code 257.6, code_mass 743).
• Correctness (midfield): verification_pct 0.83 (σ 0.10, 0.73–0.93) — on average 12.4 of 15 external checks, max 14/15, never 15/15. fable-5 is thereby above opus-4-7 (0.67) but below opus-4-6/4-8 (~0.92). The internal tests are green but do not cover the full spec: the same "tests green, spec gappy" pattern as with opus-4-6 on hybrid-v1 (F-model-novel.2) and opus-4-7 on subagents-v1 (F-model-novel.4) — but more mildly pronounced with fable-5 (no total loss of an entire operation, rather individual missing edge cases).
• Cost: mid-field at 44.5 M total_tokens (above opus-4-6's 33.9 M and 4-7's 36.1 M, below opus-4-8's 62.0 M), with the largest token spread of the RQ at σ 19.3. And with 7826 s the longest wallclock of the RQ; one run ran into the 7200 s timeout (n_ok=4/5), and two runs experienced rate-limit backoff, which inflates the wallclock further. Cost is therefore not an argument for this model here: it is neither the cheapest nor the most predictable, and it is the slowest.
• Variable cycle granularity: The five runs range from 7 to 46 RGR cycles (pred/cycle ratio consistently 2.0, markers healthy). fable-5 partly batches several it.todos per cycle instead of strictly one per test — not a quality deficit, but a difference in TDD rhythm compared to the Opus models.

Interpretation: fable-5 visibly optimizes for code quality and test discipline but "loses" some spec breadth in doing so — it implements the covered operations cleanly and fully tested but consistently omits a few spec points (never 15/15). This is the mirror image of opus-4-8 (spec-faithful up to 15/15, but higher complexity and 1.4× tokens). For quality-critical work with tolerance for ~10 % missing spec points, fable-5 is the cleanest choice — but no longer the cheapest, and the least predictable on cost; where completeness is mandatory, opus-4-8/4-6 remain ahead, and where the token budget binds, opus-4-6 leads outright.

Open follow-up question: Whether a more complete test-list step (analogous to subagents-v2 for 4-7, F-model-novel.4) lifts fable-5 from 0.83 to 4-8 correctness has not been collected — fable-5 has so far only been tested on subagents-v1.