Research Question: RQ-model-quality-oc

RQ-model-quality-oc — Findings

Setup: game-of-life-example-mapping × exact-single-context-v2-testlist-fix-oc × n=5 per cell (30 runs total, 29 without timeout). As of 2026-05-28.

Overview

Code quality as primary outcome (lower = better except where noted); correctness (verification_pct, higher = better) as gating precondition.

Trophy rule on correctness gating: Trophies for quality/efficiency metrics go only to models with verification_pct = 1.0. Kimi-K2 drops out of the pool at 0.57 mean (3 of 5 runs below 0.5) — its low cognitive_max (9.4), low lines_of_code (22.4) and short functions are partly stub artifacts (too few tests). DeepSeek-V4-Pro narrowly drops out of the pool at 0.85 — 4/5 runs are perfect, but 1 run timed out at 0.27 verification, so its low code_mass (124) and lines_of_code (24.6) do not count as clean winners. cycle_count and refactorings_applied are ambivalent (more cycles ≠ better, depending on skill compliance) — no trophy. predictions_total is exceptionally noted with a direction here: it measures compliance with the single-context-v2 skill format.

cognitive_max, cognitive_avg and longest_function: Opus and GLM are separated by less than 1 σ — spread rule → both 🏆. At longest_function Flash is exactly level with Opus (18.6) → third 🏆.

Cost calculation: per run from transcript-metrics.json.total_tokens × pricing per 1M tokens. Sources 2026-05-28: Anthropic pricing page (Opus), OpenRouter API (GLM/Kimi/DeepSeek), Vertex Standard (Gemini Flash). reasoning tokens are billed at the output rate. DeepSeek cache read is not explicitly stated on OpenRouter; conservative estimate 25 % of the input price.

Portkey gateway markup not included.

Cost breakdown per mean run (tokens as mean over n=5):

$/perfect-run scales with the share of runs with verification_pct = 1.0: Kimi $1.06/run × (5/2 verified) = $2.65 per correct solution. DeepSeek-V4-Pro $0.37/run × (5/4 verified) = $0.46. Other models 5/5 verified → unchanged.

F-1.1 — Opus 4.7 writes the most compact implementation

Opus delivers the shortest solution across all code-volume and function-size metrics (LoC, code_mass, avg/median_loc_per_function) at full external correctness and at the same time has the lowest token consumption (1.82 M vs 2.28–2.96 M for the other models with verification_pct = 1.0).

* not trophy-eligible (verification_pct < 1.0).

The median LoC/function of 3.3 (vs 7.0–14.9 for the others) means: Opus consistently extracts small helpers. At avg_loc_per_function Opus, at 7.59, is about half the size of Gemini Flash (15.63) and 40 % more compact than DeepSeek-V4-Flash (13.10).

F-1.2 — GLM 5.1 holds the Opus level in complexity

GLM 5.1 reaches values equal to or better than Opus 4.7 across smell_total, cognitive_* and mccabe_max. Both clearly beat Gemini Flash and DeepSeek-V4-Flash (cognitive-max spread 1.6–4.4 points).

GLM 5.1 thereby confirms here too (game-of-life) the impression from RQ-model-novel-oc (claim-office): near-Opus quality at OpenRouter prices. In code volume, however, Opus is clearly more compact (F-1.1).

F-1.3 — Kimi-K2 writes too few tests, fails external verification

5/5 Kimi runs report tests_passing=true, but only 2/5 reach verification_pct = 1.0 (the rest: 0.20, 0.27, 0.40). Mean tests_total is the lowest at 7.0 (vs Opus 9.4, GLM 9.8, Flash 8.4, deepseek-flash 9.2) with the highest spread (std 2.74, range 4–9).

Pattern: Kimi minimizes the test list and then implements only what its own reduced suite demands — the internal vitest runs green, but the external game-of-life-verification suite uncovers cases Kimi never tested in the first place. This is exactly the pattern claim-office-verification was originally built against: internally green tests are no proof of correctness.

F-1.4 — Gemini 3.5 Flash: fast, but the most complex code

At 153 s/run Flash is by far the fastest (Opus 231 s, deepseek-flash 612 s, GLM 835 s, Kimi 1083 s, deepseek-pro 2381 s) and reaches full correctness, but writes the most complex code: highest cognitive_max (16), cognitive_avg (14.07), mccabe_max (10.4), mccabe_avg (6.0) and the longest functions on average (avg_loc_per_function 15.63 vs Opus 7.59). Trade-off: speed against maintainability.

F-1.5 — Skill-tool compliance is model-dependent

v5.1-oc requires prediction markers per red phase. Across the six models four different compliance profiles emerge:

Where predictions are written, they are nearly 100 % correct — i.e. the marker drop is not an accuracy problem, but a question of format compliance. Confirms H3 from the RQ README: a low cycle_count (Flash 7.0) with a low predictions_total (0.4) is not "weak TDD discipline", but a failure to recognize the skill format. At the same time deepseek-v4-pro shows the highest skill compliance with 6.2 predictions_total — and also the highest cycle_count (7.6).

F-1.6 — DeepSeek-V4-Flash: cheapest path to the correct solution

DeepSeek-V4-Flash reaches full external correctness (5/5 perfect, std 0.00) at a per-run cost of ~$0.10 — an order of magnitude below the next-cheapest models (GLM-5.1 $0.74, deepseek-v4-pro $0.37, gemini-flash and kimi $1.06). At tests_total (9.2) and mccabe_avg (3.99) it sits mid-field, in code size (LoC 44.8, code_mass 189) at the upper end — the trade-off is clear: cheap + correct vs compact.

For pure correctness without a complexity requirement, Flash is 18× cheaper than Opus here. The code-volume premium (44.8 vs 38.2 LoC) is small.

F-1.7 — DeepSeek-V4-Pro: skill-compliance champion, but tail risk in duration

With predictions_total = 6.2 and cycle_count = 7.6, DeepSeek-V4-Pro produces the most structured TDD chain of all models — beyond Opus (4.8) and GLM (4.4). Prediction accuracy stays high at 90 % (5.6/6.2).

BUT: 1 of 5 runs hit the 7202 s container timeout (verification_pct = 0.27, tests_passing = false); the mean duration is the highest of all models at 2381 s and extremely variable at std=2704 s (range 854–7202 s). The other 4 runs are stable at ~1100 s with perfect verification.

The finding parallels the skill-loop issue documented for Gemini 2.5 Pro in the RQ README, but in milder form (4/5 instead of 0/3 successes). For production use Pro needs auto-recovery mechanics or a shorter timeout — in the lab setting the completed_within_budget = 80 % signal is sufficient.