Research Question: RQ-tcr-variants-sol
How do first-party EXACT Coding TCR, classic TCR, and external TCRDD variants affect correctness, code quality, development behavior, and efficiency for SOL on pi, and how do their native-Git and tool-enforced execution strategies differ?
claim-office-example-mapping
| n, run dates | Correctness | TDD discipline | Code quality | Cost | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ↑ | ↑ | ○ | ○ | ○ | ○ | ↓ | ↓ | ↓ | ↓ | ↓ | ↓ | ↓ | ↓ | ↓ | |
| baseline-inline-tdd-v1.1-local-git-piad hocn = 5, 2026-09-14 | 100% ± 0 | 100%5/5 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 7.29 ± 1.23 🏆 | 20.2 ± 3.27 | 6.60 ± 2.07 | 8.40 ± 3.58 | 0.00 ± 0.00 | 770 ± 73.72 | 4.6 ± 0.7 min 🏆 | 336k ± 65k 🏆 | $0.60 ± 0.09 🏆 |
| exact-sol-v1.3.2-local-git-control-pitest listn = 5, 2026-09-14 | 100% ± 0 | 100%5/5 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 6.80 ± 0.39 🏆 | 16.2 ± 1.30 🏆 | 4.00 ± 1.22 🏆 | 4.80 ± 1.30 🏆 | 0.00 ± 0.00 | 574 ± 34.31 | 23.1 ± 5.1 min | 5.5M ± 1.3M | $4.16 ± 0.85 |
| exact-tcr-v1-pitest listn = 5, 2026-09-14 | 100% ± 0 | 100%5/5 | 58.8 ± 7.73 | 17.2 ± 2.95 | 35.4 ± 1.95 | 5.20 ± 3.42 | 8.00 ± 2.13 🏆 | 18.2 ± 6.91 🏆 | 3.20 ± 0.84 🏆 | 4.00 ± 0.71 🏆 | 0.00 ± 0.00 | 554 ± 29.13 🏆 | 18.0 ± 2.4 min | 6.3M ± 653k | $4.54 ± 0.44 |
| external-tcr-kentbeck-2026-09-14-piad hocn = 5, 2026-09-14 | 100% ± 0 | 100%5/5 | 10.4 ± 1.95 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 8.00 ± 1.14 | 22.0 ± 6.44 🏆 | 6.80 ± 0.84 | 7.40 ± 1.14 | 0.00 ± 0.00 | 819 ± 58.79 | 7.7 ± 1.3 min | 751k ± 129k | $0.97 ± 0.12 |
| external-tcrdd-bsene-2026-09-14-piad hocn = 5, 2026-09-14 | 100% ± 0 | 100%5/5 | 1.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 9.81 ± 2.85 | 18.6 ± 2.61 🏆 | 5.40 ± 1.52 | 5.60 ± 0.89 | 0.00 ± 0.00 | 505 ± 60.32 🏆 | 11.8 ± 2.9 min | 2.3M ± 415k | $2.04 ± 0.32 |
| external-tcrdd-git-gamble-2026-09-14-piad hocn = 5, 2026-09-14 | 100% ± 0 | 100%5/5 | 24.8 ± 6.38 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 8.31 ± 1.35 | 17.4 ± 1.67 🏆 | 6.20 ± 1.79 | 6.20 ± 0.45 | 0.00 ± 0.00 | 557 ± 24.34 🏆 | 9.3 ± 1.4 min | 1.7M ± 752k | $1.61 ± 0.60 |
Runs behind this table
- 2026-09-14_13-51-49_claim-office-example-mapping_exact-sol-v1.3.2-local-git-control-pi_gpt-5-6-sol-codex
- 2026-09-14_13-51-49_claim-office-example-mapping_exact-sol-v1.3.2-local-git-control-pi_gpt-5-6-sol-codex-2
- 2026-09-14_13-51-49_claim-office-example-mapping_exact-sol-v1.3.2-local-git-control-pi_gpt-5-6-sol-codex-3
- 2026-09-14_13-51-49_claim-office-example-mapping_exact-sol-v1.3.2-local-git-control-pi_gpt-5-6-sol-codex-4
- 2026-09-14_13-51-49_claim-office-example-mapping_exact-sol-v1.3.2-local-git-control-pi_gpt-5-6-sol-codex-5
- 2026-09-14_14-09-29_claim-office-example-mapping_external-tcr-kentbeck-2026-09-14-pi_gpt-5-6-sol-codex
- 2026-09-14_14-13-45_claim-office-example-mapping_external-tcr-kentbeck-2026-09-14-pi_gpt-5-6-sol-codex
- 2026-09-14_14-13-56_claim-office-example-mapping_external-tcr-kentbeck-2026-09-14-pi_gpt-5-6-sol-codex
- 2026-09-14_14-18-00_claim-office-example-mapping_external-tcrdd-bsene-2026-09-14-pi_gpt-5-6-sol-codex
- 2026-09-14_14-20-11_claim-office-example-mapping_external-tcr-kentbeck-2026-09-14-pi_gpt-5-6-sol-codex
- 2026-09-14_14-21-35_claim-office-example-mapping_external-tcr-kentbeck-2026-09-14-pi_gpt-5-6-sol-codex
- 2026-09-14_14-21-43_claim-office-example-mapping_external-tcrdd-bsene-2026-09-14-pi_gpt-5-6-sol-codex
- 2026-09-14_14-23-44_claim-office-example-mapping_external-tcrdd-bsene-2026-09-14-pi_gpt-5-6-sol-codex
- 2026-09-14_14-28-01_claim-office-example-mapping_external-tcrdd-bsene-2026-09-14-pi_gpt-5-6-sol-codex
- 2026-09-14_14-28-42_claim-office-example-mapping_external-tcrdd-bsene-2026-09-14-pi_gpt-5-6-sol-codex
- 2026-09-14_14-31-14_claim-office-example-mapping_external-tcrdd-git-gamble-2026-09-14-pi_gpt-5-6-sol-codex
- 2026-09-14_14-32-05_claim-office-example-mapping_external-tcrdd-git-gamble-2026-09-14-pi_gpt-5-6-sol-codex
- 2026-09-14_14-38-05_claim-office-example-mapping_external-tcrdd-git-gamble-2026-09-14-pi_gpt-5-6-sol-codex
- 2026-09-14_14-39-31_claim-office-example-mapping_external-tcrdd-git-gamble-2026-09-14-pi_gpt-5-6-sol-codex
- 2026-09-14_14-40-54_claim-office-example-mapping_external-tcrdd-git-gamble-2026-09-14-pi_gpt-5-6-sol-codex
- 2026-09-14_15-44-30_claim-office-example-mapping_baseline-inline-tdd-v1.1-local-git-pi_gpt-5-6-sol-codex
- 2026-09-14_15-44-30_claim-office-example-mapping_baseline-inline-tdd-v1.1-local-git-pi_gpt-5-6-sol-codex-2
- 2026-09-14_15-44-30_claim-office-example-mapping_baseline-inline-tdd-v1.1-local-git-pi_gpt-5-6-sol-codex-4
- 2026-09-14_15-44-30_claim-office-example-mapping_baseline-inline-tdd-v1.1-local-git-pi_gpt-5-6-sol-codex-5
- 2026-09-14_15-44-30_claim-office-example-mapping_baseline-inline-tdd-v1.1-local-git-pi_gpt-5-6-sol-codex-3
- 2026-09-14_23-38-56_claim-office-example-mapping_exact-tcr-v1-pi_gpt-5-6-sol-codex
- 2026-09-14_23-38-56_claim-office-example-mapping_exact-tcr-v1-pi_gpt-5-6-sol-codex-2
- 2026-09-14_23-38-56_claim-office-example-mapping_exact-tcr-v1-pi_gpt-5-6-sol-codex-3
- 2026-09-14_23-38-56_claim-office-example-mapping_exact-tcr-v1-pi_gpt-5-6-sol-codex-4
- 2026-09-14_23-38-56_claim-office-example-mapping_exact-tcr-v1-pi_gpt-5-6-sol-codex-5
Findings
Findings — RQ-tcr-variants-sol
Overview
All six cells achieved perfect external and internal correctness. Direction: correctness and completion higher = better; complexity, Code Mass (APP), duration, and tokens lower = better. Quality and efficiency trophies are correctness-gated; every cell is eligible. Process and Git-history metrics are descriptive rather than intrinsically directional, so they receive no trophies.
| Outcome | Basic TDD | SOL Predictive TDD | First-party EXACT TCR | Classic TCR | Native-Git TCRDD | git-gamble TCRDD |
|---|---|---|---|---|---|---|
| Correctness (external) | 1.000 ± 0.000 🏆 | 1.000 ± 0.000 🏆 | 1.000 ± 0.000 🏆 | 1.000 ± 0.000 🏆 | 1.000 ± 0.000 🏆 | 1.000 ± 0.000 🏆 |
| Correctness (internal) | 100% 🏆 | 100% 🏆 | 100% 🏆 | 100% 🏆 | 100% 🏆 | 100% 🏆 |
| Completed within budget | 100% 🏆 | 100% 🏆 | 100% 🏆 | 100% 🏆 | 100% 🏆 | 100% 🏆 |
| Test blocks | 3.8 ± 0.8 | 25.2 ± 8.7 | 37.0 ± 1.6 | 7.8 ± 2.5 | 17.2 ± 1.9 | 12.2 ± 3.6 |
| Test cases total | 12.4 ± 3.2 | 29.8 ± 6.4 | 36.4 ± 2.6 | 26.0 ± 7.6 | 29.0 ± 7.7 | 19.8 ± 9.4 |
| Test cases in first block | 2.8 ± 2.5 | 1.0 ± 0.0 | 0.0 ± 0.0 | 1.8 ± 1.1 | 1.0 ± 0.0 | 0.8 ± 0.4 |
| Verified RED | 2.8 ± 1.3 | 9.8 ± 7.3 | 36.8 ± 1.8 | 6.2 ± 3.1 | 17.2 ± 1.9 | 12.2 ± 3.6 |
| Unverified RED | 1.0 ± 0.7 | 15.4 ± 2.3 | 0.2 ± 0.4 | 1.6 ± 3.1 | 0.0 ± 0.0 | 0.0 ± 0.0 |
| Explicit TCRDD refactor steps | 0.0 ± 0.0 | 0.0 ± 0.0 | 5.6 ± 4.2 | 0.0 ± 0.0 | 6.6 ± 2.6 | 2.8 ± 0.8 |
| TCR method commits | 0.0 ± 0.0 | 0.0 ± 0.0 | 58.8 ± 7.7 | 10.4 ± 1.9 | 1.0 ± 0.0 | 24.8 ± 6.4 |
| Retained RED commits | 0.0 ± 0.0 | 0.0 ± 0.0 | 17.2 ± 3.0 | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 |
| Retained GREEN commits | 0.0 ± 0.0 | 0.0 ± 0.0 | 35.4 ± 2.0 | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 |
| Retained refactor commits | 0.0 ± 0.0 | 0.0 ± 0.0 | 5.2 ± 3.4 | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 |
avg_loc_per_function ↓ |
7.29 ± 1.23 | 6.80 ± 0.39 🏆 | 8.00 ± 2.13 | 8.00 ± 1.14 | 9.81 ± 2.85 | 8.31 ± 1.35 |
longest_function ↓ |
20.2 ± 3.3 | 16.2 ± 1.3 🏆 | 18.2 ± 6.9 | 22.0 ± 6.4 | 18.6 ± 2.6 | 17.4 ± 1.7 |
cognitive_max ↓ |
6.6 ± 2.1 | 4.0 ± 1.2 | 3.2 ± 0.8 🏆 | 6.8 ± 0.8 | 5.4 ± 1.5 | 6.2 ± 1.8 |
mccabe_max ↓ |
8.4 ± 3.6 | 4.8 ± 1.3 | 4.0 ± 0.7 🏆 | 7.4 ± 1.1 | 5.6 ± 0.9 | 6.2 ± 0.4 |
| Smell Total ↓ | 0.0 ± 0.0 🏆 | 0.0 ± 0.0 🏆 | 0.0 ± 0.0 🏆 | 0.0 ± 0.0 🏆 | 0.0 ± 0.0 🏆 | 0.0 ± 0.0 🏆 |
| Code Mass (APP) ↓ | 770.2 ± 73.7 | 574.2 ± 34.3 | 553.6 ± 29.1 | 819.2 ± 58.8 | 505.0 ± 60.3 🏆 | 556.6 ± 24.3 |
| Duration ↓ | 278 ± 41 s 🏆 | 1388 ± 303 s | 1082 ± 141 s | 460 ± 77 s | 708 ± 172 s | 558 ± 84 s |
| Total tokens ↓ | 0.34 M ± 0.07 M 🏆 | 5.48 M ± 1.26 M | 6.29 M ± 0.65 M | 0.75 M ± 0.13 M | 2.32 M ± 0.41 M | 1.71 M ± 0.75 M |
The retained-commit rows describe final-history shape, not an adherence rank.
Native-Git external TCRDD deliberately squashes its phase commits at completion,
whereas first-party EXACT TCR retains phase-specific commits. RED verification
is a TDD construct and is not a fidelity criterion for Classic TCR.
test_cases_first_block does not count the first-party workflow's inactive
up-front test list as an executable test block. tcr_refactor_steps counts only
explicit TCRDD protocol executions; it does not claim that the controls
performed no structural cleanup.
F-1.7.1 — Every method delivers complete correctness
All six workflows completed within budget, passed their internal suites, and scored 1.00 on Correctness (external) in every replicate.
| Workflow | n | Correctness (external) | Correctness (internal) | Completed within budget |
|---|---|---|---|---|
| Basic TDD | 5 | 1.00 ± 0.00 | 100% | 100% |
| SOL Predictive TDD | 5 | 1.00 ± 0.00 | 100% | 100% |
| First-party EXACT TCR | 5 | 1.00 ± 0.00 | 100% | 100% |
| Classic TCR | 5 | 1.00 ± 0.00 | 100% | 100% |
| Native-Git TCRDD | 5 | 1.00 ± 0.00 | 100% | 100% |
| git-gamble TCRDD | 5 | 1.00 ± 0.00 | 100% | 100% |
On this model and kata, neither added TDD structure nor commit/revert discipline changes final behavior. The discriminating outcomes are process, structure, and efficiency rather than correctness.
F-1.7.2 — TCRDD makes RED verification and refactoring explicit
All three TCRDD variants closely coupled test blocks to verified RED states and executed explicit refactor phases. First-party EXACT TCR produced substantially more verified RED states than either external implementation.
| Workflow | Test blocks | Verified RED | Unverified RED | Explicit TCRDD refactor steps |
|---|---|---|---|---|
| Basic TDD | 3.8 ± 0.8 | 2.8 ± 1.3 | 1.0 ± 0.7 | 0.0 ± 0.0 |
| SOL Predictive TDD | 25.2 ± 8.7 | 9.8 ± 7.3 | 15.4 ± 2.3 | 0.0 ± 0.0 |
| First-party EXACT TCR | 37.0 ± 1.6 | 36.8 ± 1.8 | 0.2 ± 0.4 | 5.6 ± 4.2 |
| Classic TCR | 7.8 ± 2.5 | 6.2 ± 3.1 | 1.6 ± 3.1 | 0.0 ± 0.0 |
| Native-Git TCRDD | 17.2 ± 1.9 | 17.2 ± 1.9 | 0.0 ± 0.0 | 6.6 ± 2.6 |
| git-gamble TCRDD | 12.2 ± 3.6 | 12.2 ± 3.6 | 0.0 ± 0.0 | 2.8 ± 0.8 |
The near-zero unverified result across all three implementations points to TCRDD phase semantics rather than specifically to one enforcement tool. The first-party workflow's complete inactive test list leads to 35–39 verified RED states per run, over twice the external native-Git mean. Its explicit refactor count is variable, while both external TCRDD arms enter their prescribed refactor phase consistently. Zeroes in the other arms mean “no explicit TCRDD refactor command,” not “no refactoring.” Classic TCR is not expected to begin changes with RED and is not ranked by that criterion.
F-1.7.3 — Structured methods improve different aspects of code shape at high cost
SOL Predictive TDD has the strongest function decomposition, while first-party EXACT TCR has the lowest measured cognitive and McCabe complexity. Native-Git external TCRDD minimizes Code Mass (APP). Basic TDD and Classic TCR are cheaper but structurally weaker on most measures.
| Workflow | avg_loc_per_function |
longest_function |
cognitive_max |
mccabe_max |
Code Mass (APP) |
|---|---|---|---|---|---|
| Basic TDD | 7.29 ± 1.23 | 20.2 ± 3.3 | 6.6 ± 2.1 | 8.4 ± 3.6 | 770.2 ± 73.7 |
| SOL Predictive TDD | 6.80 ± 0.39 | 16.2 ± 1.3 | 4.0 ± 1.2 | 4.8 ± 1.3 | 574.2 ± 34.3 |
| First-party EXACT TCR | 8.00 ± 2.13 | 18.2 ± 6.9 | 3.2 ± 0.8 | 4.0 ± 0.7 | 553.6 ± 29.1 |
| Classic TCR | 8.00 ± 1.14 | 22.0 ± 6.4 | 6.8 ± 0.8 | 7.4 ± 1.1 | 819.2 ± 58.8 |
| Native-Git TCRDD | 9.81 ± 2.85 | 18.6 ± 2.6 | 5.4 ± 1.5 | 5.6 ± 0.9 | 505.0 ± 60.3 |
| git-gamble TCRDD | 8.31 ± 1.35 | 17.4 ± 1.7 | 6.2 ± 1.8 | 6.2 ± 0.4 | 556.6 ± 24.3 |
Those structural outcomes require substantially different resources. Basic TDD needs 278 seconds and 0.34 M tokens per run; Classic TCR needs 460 seconds and 0.75 M tokens; git-gamble TCRDD needs 558 seconds and 1.71 M tokens; Native-Git external TCRDD needs 708 seconds and 2.32 M tokens; first-party EXACT TCR needs 1082 seconds and 6.29 M tokens; and SOL Predictive TDD needs 1388 seconds and 5.48 M tokens. The first-party workflow is therefore faster than the SOL control but consumes the most tokens, while both are far more resource-intensive than Basic TDD.
F-1.7.4 — Composition and tooling determine the audit trail
The three TCRDD cells all preserve complete product correctness, but represent their process differently in final Git history.
| Workflow | Explicit TCRDD refactor steps | Retained method commits | Retained RED | Retained GREEN | Retained refactor commits |
|---|---|---|---|---|---|
| First-party EXACT TCR | 5.6 ± 4.2 | 58.8 ± 7.7 | 17.2 ± 3.0 | 35.4 ± 2.0 | 5.2 ± 3.4 |
| Native-Git TCRDD | 6.6 ± 2.6 | 1.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 |
| git-gamble TCRDD | 2.8 ± 0.8 | 24.8 ± 6.4 | 0.0 ± 0.0 | 0.0 ± 0.0 | 0.0 ± 0.0 |
First-party EXACT TCR retains a phase-labelled history with 48–67 method commits. The one Native-Git external TCRDD commit is prescribed final squashing, not evidence that its phase loop was skipped: tool events show 3–9 explicit refactor steps and exported reflogs retain intermediate activity. Conversely, git-gamble leaves 17–32 method commits visible while executing 2–4 explicit refactor steps; expected-RED amendments mean its final subjects are not classified as retained RED/GREEN/refactor commits by the history metric. Auditability must therefore be assessed against each composition's intended representation rather than by comparing final commit counts as a universal adherence score.