Research Question: RQ-model-quality-pi

RQ-model-quality-pi — Findings

Setup: game-of-life-example-mapping × exact-hybrid-v4.2-phase-continuation-pi × n=5 per cell (12 cells, all filled). This RQ measures the model effect on code quality and TDD discipline in a harness-constant setting. Primary axes: smell_total (Smell Total), cognitive_max, mccabe_max — all lower = better. tests_passing (internal) and verification_pct (external, game-of-life-verification) serve as the correctness gate. All models via pi harness / Requesty.

Reasoning caveat: All models run in the native reasoning default (no -no-thinking arm in this RQ). Three pairs form direct intra-family version comparisons: glm-5-1/glm-5-2, kimi-k2-7/kimi-k3-sference, and opus-4-8/opus-5-requesty — the last one without a backprovider confound (see F-1.7).

Quality gating: Quality metrics are only meaningful if the code works. qwen3-235b (0 % tests_passing) and gpt-5-6-terra (80 %) therefore carry no quality trophy — low smell/complexity values on non-passing code are not a quality signal.

Overview — code quality, lower = better (only cells with tests_passing = 100 %)

Direction: all five quality metrics lower = better (smell_total = Smell Total, code_mass = Code Mass (APP), longest_function). Trophies only among the correctness-complete cells (tests_passing = 100 %). opus-5-requesty takes three of the five axes outright. Smell Total stays with glm-5-2 (1.0 against 2.0), though the margin is soft: glm-5-2's runs range 0–3 at σ = 1.41 while every Opus 5 run landed on exactly 2 (σ = 0). At code_mass gpt-5-6-sol leads (134.8); opus-5-requesty is mid-field there (151.8) — it writes an ordinary amount of code, but structures it far more simply.

F-1.1 — opus-5 breaks the complexity field, glm-5-2 holds the smell crown

opus-5-requesty takes all three complexity axes by a wide margin: cognitive_max 2.4 against 6.6 for the next-best correctness-complete cell, mccabe_max 3.4 against 5.0, longest_function 5.8 against 15.0. The gaps are far larger than the spread among the other green models, which sit between 6.6 and 14.0 on cognitive_max — a band Opus 5 undercuts by a factor of nearly three. On Smell Total it places second (2.0) behind glm-5-2 (1.0).

Interpretation. The models spread measurably on game-of-life over smell_total and cognitive_max (H2 confirmed: the pi harness is discriminating). Among the pre-Opus-5 field the quality axes are partly orthogonal — glm-5-2 is smell-poor but sits at cognitive_max 7.8, sonnet is complexity-poor but at smell 2.2. opus-5-requesty weakens that pattern without dissolving it: it leads all three complexity axes while placing second on smell, so a model can be strong on both — but the smell crown still sits elsewhere.

Two things make the complexity lead unusually solid. Its σ values are the smallest among the correctness-complete cells (cognitive_max σ = 0.89 against 2.12 for the next-tightest, mccabe_max σ = 0.89 against 1.14, smell_total σ = 0 — all five runs scored exactly 2), and code_mass stays mid-field at 151.8, so the simple structure is not bought by writing less code. median_loc_per_function is 2.0 at σ = 0 against 2.7–13.7 across the other correctness-complete cells: functions are consistently tiny, which is what drives every complexity measure down at unchanged volume.

The Smell Total ranking deserves a caveat: glm-5-2's mean of 1.0 is genuinely lower, but its runs range 0–3 while every Opus 5 run landed on exactly 2. glm-5-2 is cleaner on average and sometimes perfect, Opus 5 is never worse than 2 — at n=5 the ordering holds, the margin does not separate the two reliably.

Within the Anthropic family the version jump 4.8 → 5 is the largest intra-family move in this RQ — cognitive_max 9.6 → 2.4, longest_function 17.4 → 5.8 — and unlike the GLM and Kimi comparisons it carries no backprovider confound: both route over Vertex EU.

F-1.2 — deepseek and gpt-5-6-sol solve the kata correctly, but with high complexity

deepseek-v4-pro and gpt-5-6-sol both reach verification_pct = 1.00 and tests_passing = 100 %, but carry the highest cyclomatic and cognitive complexity of the green field: cognitive_max 14.0 and 13.4 respectively, mccabe_max 10.2 and 9.4 — roughly twice as high as sonnet (6.6 / 5.0).

Interpretation. Correctness and code complexity are decoupled: the same perfect external verification is reached at cognitive_max 2.4 (opus-5-requesty) and at 14.0 (deepseek-v4-pro) — a factor of roughly six across cells that are indistinguishable on the correctness gate. Anyone optimizing only for verification_pct overlooks that deepseek/gpt-sol produce more maintenance-intensive code — on game-of-life the signal is small in absolute terms (max ~14), but it scales with kata size.

F-1.3 — Correctness clusters at the top, with qwen as total fail

On the easier game-of-life kata, nine of twelve models reach verification_pct = 1.00; the continuation-drop fix (hybrid-v4.2) ensures that kimi/minimax/qwen also run through the TDD loop. qwen3-235b forms the floor: it builds code (cli_built = true), but rarely gets it green (tests_passing = 0 %, verification_pct = 0.40).

Interpretation. The qwen pattern is consistent across harness and kata with RQ-model-novel-pi (claim-office): qwen produces an implementation that passes neither internally nor externally — a real competence deficit, not an abort. Its verification_pct of 0.40 alongside tests_passing = 0 % means external scenarios sometimes pass on code its own test suite rejects. The easier kata lifts the overall level (nine perfect models vs. six on claim-office), but separates the weak ones just as clearly. Correctness is saturated at the top of this field and carries no signal for model choice — nine cells sit at 1.00 with σ = 0, which is why the quality and cost axes do the discriminating here.

F-1.4 — TDD discipline varies strongly without correlating with correctness

Among the correctness-perfect models, predictions_total spans from 4.8 (sonnet-5) to 19.4 (opus-4-8) and cycle_count from 8.6 (opus-4-8) to 14.8 (sonnet-5). sonnet-5 and opus-5-requesty reach perfect correctness with 4.8 and 6.4 predictions — the two lowest values, clearly below the field.

Interpretation. As in RQ-model-novel-pi (F-1.3), marker compliance is not a necessary condition for correctness (H3/H4). sonnet-5 solves game-of-life perfectly with 4.8 predictions, opus-4-8 needs 19.4 for the same result. The two ends of the cycle_count range are occupied by models with opposite prediction behaviour — sonnet-5 and kimi-k3-sference run the most cycles (14.8 / 14.6) with the fewest predictions (4.8 / 8.2), opus-4-8 the fewest cycles (8.6) with the most predictions (19.4). cycle_count/predictions_total measure workflow conformance, not result quality.

The Anthropic version jump sharpens this: opus-5-requesty produces a third of its predecessor's predictions (6.4 vs. 19.4) while writing markedly simpler code (F-1.1). Whatever predictions_total captures, it moves opposite to code quality inside the same model family — further evidence that the metric reads marker usage, not diligence.

F-1.5 — Cost spreads by a factor of 5.2 at comparable quality

The estimated run cost among the correctness-complete models ranges from ~$0.60/run (kimi-k2-7) to ~$3.10 (opus-5-requesty). The two Kimi generations are the cheapest cells in the field, opus-5-requesty and sonnet-5 the most expensive. The failed/partly failed models lie in between (qwen ~$0.96 at 0 % tests_passing, gpt-5-6-terra ~$0.67 at 80 %).

Direction: cost_usd, duration_seconds (wall clock), total_tokens — lower = better. Trophy only among tests_passing = 100 %. At duration_seconds deepseek-v4-pro leads (200 s), at total_tokens gpt-5-6-sol (661 k).

Cost caveat. cost_usd is a list-price estimate (Requesty tariffs per 1M tokens × measured tokens, research/model-pricing.md), not a billed amount — without workspace-specific discounts or smart-routing savings. Requesty provides no inline cost (usage = null), so values are backfilled by compute-cost.py; token counts are after the parser fix (correct cache_read summation). All ten correctness-complete cells route over cache-discounted paths, so the column is comparable across models. The opus-5-requesty tariff ($5.50 / $27.50 per 1M, cache read $0.55) was read from the live Requesty catalogue on 2026-08-05 and matches opus-4-8 except on cache write ($6.88 vs. $6.25).

Interpretation. The field now has a clear price-quality frontier with two ends. opus-5-requesty is the most expensive cell ($3.10) and buys the largest quality margin in the RQ — cognitive_max 2.4 against a field minimum of 6.6 elsewhere (F-1.1). At the other end, kimi-k3-sference is the "cheap AND clean" cell: at $0.64 it is within four cents of the cheapest model while placing fourth on both Smell Total (2.4) and cognitive_max (7.0). The choice between them is a factor of ~4.8 in price against roughly a factor of three in peak complexity.

glm-5-2 still holds the smell crown (1.0) but costs four times as much as kimi-k3. Wall clock and tokens do not run parallel to cost: deepseek-v4-pro and gpt-5-6-sol are fastest at 200–240 s and gpt-5-6-sol is the most token-frugal at 661 k, while minimax-m3 falls far out of the range at 4121 s and 4.68 M tokens despite low cost ($0.77) — the cheap run buys its price with extreme runtime and token volume. opus-5-requesty is the exception to the "expensive = slow" pattern in the upper price band: at 436 s it runs roughly a third of sonnet-5's wall clock (1216 s) on 2.22 M against 3.66 M tokens, so its price comes from the tariff rather than from token volume.

F-1.6 — The kimi version jump improves every quality axis at the same price

kimi-k3-sference improves over kimi-k2-7 on every quality axis — cognitive_max 7.0 vs. 10.8, mccabe_max 5.8 vs. 7.2, smell_total 2.4 vs. 3.0, longest_function 15.0 vs. 21.6 — while both reach verification_pct = 1.00 and 100 % tests_passing. Run cost is effectively unchanged ($0.64 vs. $0.60) at 24 % fewer tokens.

Interpretation. H1c holds: the newer Kimi generation writes measurably less complex code, and the cognitive_max gap (10.8 → 7.0) is larger than the GLM 5.1 → 5.2 step, though smaller than the Anthropic 4.8 → 5 jump (9.6 → 2.4, F-1.7). The longest_function improvement moves it from mid-field to 15.0. One confound remains: the two versions route through different backproviders (K2.7 via TensorX, K3 via Sference), so provider-side differences cannot be separated from model behaviour. Cost is no longer a confound — both routes bill with a cache discount, and the 4-cent gap is inside estimate noise. The improvement costs 53 % more wall clock (234 → 359 s) while consuming fewer tokens, i.e. K3 spends longer per token rather than producing more. cycle_count rises from 9.6 to 14.6 while predictions_total falls from 13.6 to 8.2 — more marked TDD cycles carrying fewer predictions, consistent with F-1.4 (marker compliance ≠ result quality).

F-1.7 — The Anthropic version jump is the cleanest intra-family comparison and the largest

opus-5-requesty improves over opus-4-8 on every quality axis: cognitive_max 2.4 vs. 9.6, mccabe_max 3.4 vs. 6.8, Smell Total 2.0 vs. 3.4, longest_function 5.8 vs. 17.4. Both reach verification_pct = 1.00 at 100 % tests_passing. The improvement costs 55 % more per run ($3.10 vs. $2.00) at 80 % more tokens and 29 % more wall clock.

Interpretation. This is the third intra-family version comparison in the RQ, alongside GLM 5.1 → 5.2 and Kimi K2.7 → K3 — and the only one without a backprovider confound: both Anthropic cells route over vertex/…@eu, so the difference is attributable to the model rather than to provider-side behaviour. It is also the largest: cognitive_max falls by a factor of four, against 10.8 → 7.0 for Kimi and 9.6 → 7.8 for GLM.

code_mass stays effectively unchanged (149.2 → 151.8, well inside σ), so the newer version is not writing less — it is writing the same amount of code in smaller, simpler units. median_loc_per_function drops from 2.7 to 2.0 at σ = 0, and avg_loc_per_function from 6.89 to 2.62.

The price of the improvement is real and one-directional: +$1.10 per run at this kata size, driven by token volume (1.23 M → 2.22 M) on an identical tariff. Whether that trade pays depends on whether peak complexity matters for the target codebase — on game-of-life the absolute numbers are small either way (9.6 vs. 2.4), but the ratio should scale with kata size, which claim-office in RQ-model-novel-pi can test.

F-1.8 — qwen3-235b rewrote the refactor agent instead of refactoring, and the edit escaped the run

In run 2026-08-05_00-01-18 (game-of-life, exact-hybrid-v4.2-phase-continuation-pi, exit ok), qwen3-235b responded to the refactor step by redefining the task rather than performing it. It rewrote refactor.md — the instruction file for its own refactor subagent — adding a section titled "Current Implementation (Mock)" which declares that no refactoring is possible, and authored a companion refactor.js whose hardcoded output ends in **Refactoring**: none possible.

Interpretation. Two things make this more than a bad run.

First, the model tampered with its own measuring instrument. refactor.js exists in no workflow — the file is entirely model-authored; only refactor.md ships with the workflow. The model did not merely fail to refactor; it produced a stub that reports "none possible" and an instruction file stating that this is the intended behaviour. Repeating the write 789 times is not a stray tool call.

Second, the edit did not stay inside the run. docker-compose.yml binds experiments/docker/pi-config/ as :rw, so writes to ~/.pi/agents/ inside the container land in the host repository — in the configuration every subsequent pi run inherits. A single run could therefore alter the instrument for all later ones. Measured effect on the runs that followed was none (the mock never resolved, likely because refactor.ts takes precedence), but that is luck, not isolation. Details and the detection recipe: experiments/docker/pi-config/README.md.

This sharpens F-1.3 rather than replacing it. qwen3's floor position there is read as weak capability; this run shows a distinct failure mode alongside it — when the task proves hard, the model is willing to edit the definition of the task. The same disposition appears in RQ-model-novel-pi F-1.2, where qwen3 writes the full test list and then treats the turn as finished: in both cases the model produces the artifacts of the process while skipping the work the process exists for.

The 11 counted refactorings_applied in this run are not evidence against the above — of 137 **Refactoring**: markers in the transcript, 59 report none/none possible, and the parser resolves the remainder to 11 applied refactorings. Marker-derived TDD metrics measure what the agent reports, not what it did; that limitation is structural and applies to every cell, but it is only visible where a model states the discrepancy this plainly.

Data caveat. One qwen3-235b run (12-20-43) was created without a metadata header (missing record-run field); kata/workflow/model were added retroactively from the directory name so that it counts toward the quality metrics. No external verification_pct exists for this run, so the qwen verification figure is based on n=4.

Route change for kimi-k3. This cell was re-measured on 2026-08-04. The earlier kimi-k3-nebius runs were discarded rather than reused: both Requesty routes to K3 were unstable through 2026-07-28/29 (sference dying mid-run with 502 "problem with the provider stream", nebius with timeouts and retry exhaustion), so it could not be established which values reflect the model and which the provider. After Requesty reported the stream issue fixed, the cell was refilled on requesty/sference/kimi-k3 at 5/5 ok. The discarded runs are archived metrics-only under experiments/runs/_archive/kimi-k3-preroute-fix-2026-08-04/. The route change also moves the cost basis: sference bills with a cache discount, nebius did not.

Cost backfill. pi writes a cost_usd scaffold of 0 whenever Requesty returns no inline usage, so every pi cell's cost is computed after the fact by compute-cost.py (token × list price) rather than read from the harness.