Research Question: RQ-tdd-correctness

RQ-tdd-correctness Findings

Kata: claim-office-example-mapping (novel). Model: opus-4-7-no-thinking (Portkey OR direct, OR-match). 5 TDD workflow variants, n=22 runs. end-refactor-only-v1-agent/end-refactor-only-v1-native (non-TDD control group) still without runs.

Overview — Correctness per Workflow

🏆 = best value per column (also multiple times in case of a tie). verification_pct/tests_passing: higher = better.

completed_within_budget is 100 % in all cells.

F-tdd-correctness.1 — Three of Five TDD Workflows Solve claim-office Perfectly; subagents-v2 and green-refactor-v2 Lose Isolated Scenarios

On the novel claim-office kata that is not contained in the training data, inline-tdd-v1 (n=5), single-context-v2 (n=6) and hybrid-v2 (n=3) reach the full acceptance suite (15/15 verification scenarios) in every single run. subagents-v2 shows one outlier (4/5 runs perfect, 1 run at 12/15 → 0.96), green-refactor-v2 a smaller one (2/3 runs perfect, 1 run at 14/15 → 0.98). Notably: the two workflows with an isolated green subagent (subagents-v2, green-refactor-v2) each carry one correctness outlier; the three workflows with green in the shared context (inline-tdd-v1, single-context-v2, hybrid-v2) are perfect. Plausible mechanic: an isolated green subagent sees neither the test-list discussion nor earlier cycle discussions and can overlook edge cases that are implicitly present in the shared context.

Hypothesis H1 ("phase-structured workflows reach higher correctness than minimal TDD") is therefore not confirmed — minimal TDD (inline-tdd-v1) is on a par with the structured workflows. H3 (null hypothesis: all workflows similarly high >0.8) is confirmed.

F-tdd-correctness.2 — subagents-v2 Reaches Correctness Only via Drastically Higher Effort per Cycle

The effort profiles per workflow. 🏆 = best value per column. Directions: predictions_correct_rate higher = better; duration_seconds/total_tokens/tests_passed_immediately lower = better; cycle_count and refactorings_applied are ambivalent (no trophy).

subagents-v2 runs on average 44.6 TDD cycles per run (vs. 3.8 for inline-tdd-v1, 5.5 for single-context-v2), at comparable correctness. The wallclock is at ~54 min per run against ~10 min for single-context-v2 and ~5 min for inline-tdd-v1. Tokens 34.9 M (subagents-v2) vs. 3.3 M (inline-tdd-v1), a factor of 10.6. Despite this effort, subagents-v2 is the only setup with a 0.8 outlier.

The token column counts subagent consumption; until 2026-09-30 it did not, and the three isolating arms here were the ones it understated — subagents-v2 most of all, at 14.10 M against a true 34.87 M (RQ-old-vs-new-exact-line-opus55 F-4.12.5). The finding's own point is sharpened by it: the effort subagents-v2 spends for its correctness is 2.5× what was recorded.

hybrid-v2 and green-refactor-v2 both run considerably more refactor steps than single-context-v2 (11.0 / 14.0 vs. 2.2) — the isolated refactor subagent visibly "works" more; green-refactor-v2 reaches the highest refactor rate of all workflows. Despite the hybrid constructions, both pay for stability, not for a better correctness mean.

F-tdd-correctness.3 — The Predictions Rate Comparison Is Distorted by an Unequal Prediction Base

predictions_correct_rate is at 100 % for single-context-v2 (39/39) and green-refactor-v2 (99/99), 94.9 % for hybrid-v2 (131/138), 92.9 % for subagents-v2 (302/325). The denominators differ drastically: single-context-v2 ~6.5, green-refactor-v2 ~33, hybrid-v2 ~46, subagents-v2 ~65 predictions/run. The predictions rate here does not primarily measure discipline but is dominated by the task size per cycle — subagents-v2 decomposes most finely and uses more predictions, thereby also having more opportunities for errors.

Hypothesis H3 from RQ-tdd-quality ("subagents-v2 has higher prediction_accuracy") is not confirmed under this reading. The comparison only becomes robust once the predictions are normalized per cycle — currently not directly derivable from the metrics.

F-tdd-correctness.4 — The Wallclock Range Is 10×, the Token Range 12×; No Correlation with Correctness

Across the five TDD workflows:

• cheapest workflow by tokens: inline-tdd-v1 (3.28 M) — at 100 % correctness
• cheapest workflow by wallclock: inline-tdd-v1 (5 min)
• most expensive workflow by tokens: hybrid-v2 (37.88 M, σ=13.5 M) — at 100 % correctness
• most expensive workflow by wallclock: subagents-v2 (54 min, σ=15 min) — at 0.96 correctness

The three arms that isolate a phase are a token tie at the top of the range (34.87 / 35.74 / 37.88 M, every pairwise gap inside its σ), so the range is really two steps rather than five: 3.3–3.6 M without isolation, 18.7 M with a shared context, and ~35 M once any phase is isolated. green-refactor-v2 slots in on wallclock (33 min) but not on tokens (35.74 M, level with the other isolating arms). For claim-office under Opus 4.7, correctness is not a scarce good; the workflow choice determines almost exclusively effort and spread. inline-tdd-v1 dominates the correctness-per-token rating on this kata. Structured workflows do not justify themselves on claim-office through correctness — their value lies in code quality (see RQ-context F-context.1/2 for the complexity and smell differences on the same kata).