Agentic Coding LabData as of 2026-10-16 00:24 UTC, 1,273 runs, commit 4af5566

What the lab measured, one research question at a time

Each question fixes some factors, varies others, and runs coding agents against a kata until every cell has enough replicates. The tables are computed from the runs.

Claude Code

Does example mapping increase correctness compared to prose and user story — and is the effect model-dependent?RQ-prompt-correctness
22/24 cells, 126 runs, 2026-05-08 → 2026-06-02
Does the prompt style (prose/user-story/example-mapping) influence correctness and code quality on a training-known kata (Game of Life) — and is this effect model-dependent?RQ-prompt-known-kata
9/9 cells, 45 runs, 2026-05-12 → 2026-05-13
How strongly do the available models (Sonnet 4.6, Opus 4.6, Opus 4.7, Opus 4.8, Fable 5 — each with/without thinking) differ in code quality on a training-known kata under the strongest workflow?RQ-model-quality
12/12 cells, 44 runs, 2026-05-14 → 2026-07-25
How do Fable 5, Opus 4.8, Opus 4.7 and Opus 4.6 (each no-thinking) differ in correctness and code quality on a novel kata with ambiguities that differentiates more strongly than the training-known game-of-life?RQ-model-novel
2/5 cells, 27 runs, 2026-05-10 → 2026-07-26
How do Fable 5, Fable 5.1, Opus 5 and Sonnet 5 (each no-thinking, on the Claude Max subscription) differ in correctness and code quality on the novel claim-office kata under the current exact-coding baseline workflow?RQ-fable-vs-opus5
4/4 cells, 20 runs, 2026-09-10 → 2026-09-11
Does the maintained EXACT Coding Predictive TDD workflow still change code quality on Opus 5.5 the way it does on Opus 5, and does Opus 5.5 still need it at all compared with a minimal inline-TDD instruction?RQ-opus55-current-workflow
5/6 cells, 28 runs, 2026-10-01 → 2026-10-01
Does the quality of a TDD workflow depend on the model — is there a universally best workflow, or do different workflows swap places depending on the model?RQ-workflow-model
5/6 cells, 48 runs, 2026-05-08 → 2026-05-19
How does the workflow structure (from oneshot through iterative to strict TDD with subagents) affect code quality, and does TDD strictness make a difference?RQ-tdd-quality
16/16 cells, 103 runs, 2026-05-08 → 2026-05-25
Does external correctness (verification_pct) differ between TDD workflow variants on the novel claim-office kata?RQ-tdd-correctness
7/7 cells, 36 runs, 2026-05-08 → 2026-05-25
Which form of context structuring — isolated subagent contexts per TDD phase (subagents-v2), a shared, accumulated single context (single-context-v2), a hybrid with skill-based red/green in the shared context and an isolated refactor subagent (hybrid-v2), or a hybrid with isolated green and refactor subagents alongside a shared-context test list/red (green-refactor-v2) — leads to better code quality?RQ-context
4/4 cells, 21 runs, 2026-05-20 → 2026-05-24
Does the TDD architecture axis (inline-tdd-v1 structureless / single-context-v2 single context / hybrid-v2 hybrid / hybrid-v6 current generation) still rank the same way on opus-5 as it does on opus-4-7 — and does the decomposition metric change the answer?RQ-architecture-axis-opus5
16/16 cells, 111 runs, 2026-05-08 → 2026-09-10
Is the native Sol workflow line (basic-sol-tdd, Predictive TDD, Four Rules refactor) better only on Sol/pi, or does it also beat the opus line on Opus with native Claude Code — and does the APP effect that suppresses decomposition on Sol reproduce there?RQ-sol-line-on-opus-cc
5/6 cells, 30 runs, 2026-05-29 → 2026-08-17
Can the inner TDD loop of EXACT Coding be substituted by an externally authored TDD workflow, and what does the substitution cost or buy? Example mapping stays the entry point; only the implement/test/refactor loop is exchanged. Measured on claim-office-example-mapping against the current exact-coding baseline exact-hybrid-v2.4-lab-split-cc — on correctness, code quality, TDD discipline and cost.RQ-external-tdd-workflows-opus5
3/4 cells, 15 runs, 2026-09-03 → 2026-09-04
How do classic TCR and TCRDD variants affect correctness, code quality, development behavior, and efficiency in autonomous coding agents, and—within TCRDD—does tool-enforced execution with git-gamble differ from native-Git prompt enforcement?RQ-tcr-variants-opus5
4/4 cells, 18 runs, 2026-09-14 → 2026-09-14
On native Opus 5, how does the current SOL Predictive-TDD workflow compare with the default EXACT Coding workflow on correctness, decomposition, development behavior, and efficiency?RQ-current-ptdd-vs-exact-opus-native
2/2 cells, 28 runs, 2026-08-10 → 2026-09-16
On native Opus 5, does an independent-dimensions cross-check in the SOL PTDD test-list phase improve external completeness without sacrificing the workflow's efficiency advantage or code quality?RQ-test-list-dimensions-opus-native
2/2 cells, 20 runs, 2026-09-15 → 2026-09-16
On native Opus 5, can naming-first and a mandatory inline refactoring trial improve SOL PTDD v1.6 decomposition without paying the isolated-subagent cost of the Opus default?RQ-inline-refactor-operationalization-opus-native
2/2 cells, 15 runs, 2026-09-16 → 2026-09-16
On Opus 5.5, does the retired EXACT Coding Opus line still decompose more strongly than the maintained Predictive-TDD line, and does isolating the Refactor step into a subagent help or hurt within each line?RQ-old-vs-new-exact-line-opus55
4/4 cells, 20 runs, 2026-09-30 → 2026-10-01
How do seven TDD workflows compare on correctness, TDD discipline, code quality and cost? Three lines meet on one kata: the maintained EXACT Coding Predictive-TDD pair (with and without an isolated refactor subagent), the retired Opus hybrid pair, and three vendored third-party TDD skills — measured on claim-office-example-mapping × opus-5-5-no-thinking × Claude Code.RQ-tdd-workflow-comparison-opus55
7/7 cells, 35 runs, 2026-09-30 → 2026-10-01
How stable are code quality and TDD discipline per workflow across replicates, and under which conditions is n=3 a sufficient replicate count?RQ-stability
5/6 cells, 59 runs, 2026-05-14 → 2026-05-15

pi

How do the models reachable via the pi harness (Requesty routing) differ in code quality and TDD discipline on game-of-life-example-mapping with the exact-hybrid-v4.2-phase-continuation-pi workflow?RQ-model-quality-pi
12/12 cells, 60 runs, 2026-07-25 → 2026-08-05
How do the models reachable via the pi harness (Requesty routing) differ in correctness and TDD discipline on claim-office-example-mapping with the exact-hybrid-v4-cleaned-pi workflow?RQ-model-novel-pi
16/18 cells, 86 runs, 2026-07-24 → 2026-08-05
How do transport route and reasoning channel each affect code quality, TDD discipline, throughput and correctness for one and the same model (GPT-5.6 Sol), at constant harness, workflow, kata and prompt style?RQ-route-effect-pi
3/3 cells, 15 runs, 2026-08-15 → 2026-08-16
On the OpenAI subscription route, how does GPT-5.3 Codex Spark compare to GPT-5.6 Sol on the two native Sol workflows — does the smaller, cheaper-tier model hold correctness and code quality, or does it only look competitive because the workflow carries it?RQ-spark-vs-sol
4/4 cells, 20 runs, 2026-08-16 → 2026-08-17
On the OpenAI subscription route, how does GPT-6 Astra compare to the strongest references reachable on the pi harness — Sol on the same route, Sol on Requesty in both reasoning states, and Opus 5 on Requesty — at constant harness, workflow, kata and prompt style?RQ-astra-pi
5/5 cells, 25 runs, 2026-08-05 → 2026-09-05
Does the natively-built Sol workflow line (basic-sol-tdd, Predictive TDD) transfer to GPT-6 Astra — or is its advantage over the Opus-derived EXACT line a property of Sol?RQ-astra-native-sol
8/8 cells, 40 runs, 2026-08-16 → 2026-09-13
How do first-party EXACT Coding TCR, classic TCR, and external TCRDD variants affect correctness, code quality, development behavior, and efficiency for SOL on pi, and how do their native-Git and tool-enforced execution strategies differ?RQ-tcr-variants-sol
6/6 cells, 30 runs, 2026-09-14 → 2026-09-14
On GPT-5.6 SOL via pi, does the independent-dimensions test-list cross-check improve the current SOL PTDD default's external completeness without sacrificing its efficiency or product quality?RQ-test-list-dimensions-sol-pi
2/2 cells, 20 runs, 2026-09-15 → 2026-09-16
Across Game of Life and Sphinx Score on GPT-5.6 SOL/pi, does the v1.6 independent-dimensions test-list cross-check justify its overhead relative to the v1.5 SOL default?RQ-test-list-dimensions-multikata-sol-pi
4/4 cells, 20 runs, 2026-09-16 → 2026-09-16
Does GPT-6 Sol match or beat GPT-5.6 Sol on the canonical Predictive TDD workflow (exact-ptdd-v1-pi) — in correctness, code quality and cost?RQ-gpt6-sol-vs-gpt56-sol
4/4 cells, 20 runs, 2026-09-23 → 2026-09-23
On GPT-6 Sol, what does EXACT Coding Predictive TDD buy over a plain inline-TDD baseline, and does isolating the Refactor phase in a subagent add anything on top?RQ-workflow-axis-gpt6-sol
6/6 cells, 30 runs, 2026-09-23 → 2026-09-24
On GPT-6.1 Sol, what does EXACT Coding Predictive TDD buy over a plain inline-TDD baseline, and does isolating the Refactor phase in a subagent add anything on top?RQ-workflow-axis-gpt61-sol
6/6 cells, 30 runs, 2026-10-04 → 2026-10-05

OpenCode

How do five models reachable via the OpenCode harness (Opus 4.7 via Portkey + four non-Anthropic models from the Portkey catalog) differ in code quality and TDD discipline on game-of-life-example-mapping with the exact-single-context-v2-testlist-fix-oc workflow?RQ-model-quality-oc
5/6 cells, 29 runs, 2026-05-25 → 2026-05-28
How do five models reachable via the OpenCode harness differ in correctness and TDD discipline on claim-office-example-mapping with the exact-single-context-v2-testlist-fix-oc workflow?RQ-model-novel-oc
6/8 cells, 38 runs, 2026-05-25 → 2026-05-28

Cursor CLI

How do the models reachable via the cursor-cli harness (Opus 4.8 medium, Composer 2.5, Grok 4.5 medium) differ in code quality and TDD discipline on game-of-life-example-mapping?RQ-model-quality-cursor
2/3 cells, 10 runs, 2026-07-28 → 2026-07-28

Across harnesses

How does switching harness (Claude Code vs OpenCode vs pi) affect correctness, code quality, TDD discipline and cost when model (opus-4-8 via Requesty), workflow intention and prompt style are held constant?RQ-harness-requesty
6/6 cells, 30 runs, 2026-07-24 → 2026-07-25
How much cheaper is the GPT model gpt-5-6-sol on the pi harness compared to opus-4-8 on Claude Code — at the same prompt style and an outcome-equivalent TDD workflow, across both katas?RQ-cost-sol-pi-vs-opus-cc
4/4 cells, 20 runs, 2026-07-25 → 2026-07-25
Does the code-quality profile of Opus (opus-4-8) differ between the Claude Code and the pi harness, each with and without thinking, at a constant workflow generation (hybrid-v4)?RQ-model-quality-cc-vs-pi
4/4 cells, 20 runs, 2026-05-30 → 2026-07-26
Under a bare 'use TDD' instruction that prescribes no phase markers (inline-tdd-v1), do models actually work test-first and refactor — and how far apart do the models sit once the evidence is hand-validated?RQ-v3-emergent-tdd
10/10 cells, 49 runs, 2026-05-08 → 2026-08-11
Does the apparent platform interaction between PTDD v1.5 and v1.6 replicate at n=10 per cell on Claim Office, or was it driven by small-sample variance?RQ-test-list-dimensions-replication
4/4 cells, 40 runs, 2026-09-15 → 2026-09-16
Does isolating only the per-cycle Refactor phase improve PTDD v1 product structure enough to justify its context cost on native Opus and GPT-5.6 SOL/pi?RQ-ptdd-refactor-subagent-cross-model
4/4 cells, 42 runs, 2026-09-16 → 2026-10-01
On Java 17 with JUnit 5 and Maven, how do inline TDD, shared-context EXACT Coding Predictive TDD, and EXACT Coding with an isolated Refactor subagent compare on correctness and code quality for GPT-5.6 SOL and Opus 5?RQ-exact-coding-java
11/12 cells, 59 runs, 2026-09-20 → 2026-09-20
On Python with pytest, how do inline TDD, shared-context EXACT Coding Predictive TDD, and EXACT Coding with an isolated Refactor subagent compare on correctness and code quality for GPT-5.6 SOL and Opus 5?RQ-exact-coding-python
12/12 cells, 60 runs, 2026-09-21 → 2026-09-22
On TypeScript with Vitest, how do inline TDD, shared-context EXACT Coding Predictive TDD, and EXACT Coding with an isolated Refactor subagent compare on correctness and code quality for GPT-5.6 SOL and Opus 5?RQ-exact-coding-typescript
6/6 cells, 57 runs, 2026-09-14 → 2026-10-01
On Rust with Cargo, how do inline TDD, shared-context EXACT Coding Predictive TDD, and EXACT Coding with an isolated Refactor subagent compare on correctness, TDD discipline and code quality for GPT-6 SOL and Opus 5?RQ-exact-coding-rust
5/6 cells, 29 runs, 2026-10-01 → 2026-10-01
Does splitting the PTDD v1 test list into driving tests and a trailing verification group keep Correctness, safety net and code quality at least at the v1 level on Opus 5.5 and GPT-6.1 Sol, while making the phase-chain discipline readable for a test-list workflow?RQ-ptdd-verification-split
2/4 cells, 11 runs, 2026-10-01 → 2026-10-06