Research Question: RQ-astra-native-sol

Findings — RQ-astra-native-sol

Overview

Claim Office, example-mapping, pi harness and OpenAI subscription route. Eight cells × five runs = 40 runs. Floor = baseline-inline-tdd-v1-pi; Inline = exact-sol-v1-pi; Isolated = exact-sol-v1.1-subagent-pi; EXACT = exact-hybrid-v4.2-phase-continuation-pi. Astra = gpt-6-astra-codex-no-thinking; Sol = gpt-5-6-sol-codex, with the README's documented equivalent labels. All cells have 100% internal correctness and completion within budget.

Astra

Sol

Means; all sample SDs and ranges appear in summary.md. Trophies compare workflows within each model, not an implicit eight-cell contest. Only clear contrasts are marked; ties at a universal ceiling are not crowned. All cells meet the standard 0.90 correctness gate, including Sol Isolated, though its failure matters for the RQ's stricter recommendation criterion. Code Mass, function counts and marker counts are not quality rankings. Low Production LoC can accompany very weak decomposition, so it carries no trophy here.

Cohort limitation: Astra's cells are September 5 runs; Sol Inline consists of September 13 fresh-comparison runs, while the remaining Sol cells are August runs. Date/environment confounds all cross-model contrasts and also Sol's within-model Inline contrasts. Astra's within-model comparisons retain their shared cohort. No new experiments or analysis-pipeline reruns were performed for this update; existing metrics were reaggregated and costs recomputed.

F-1.6.1 — The native inline arm has better observed quality means than EXACT

Correctness is perfect in all four cells. The Astra cognitive-peak difference (2.0, SDs 0.45 and 0.84) is clearer than its noisy function-length contrast (EXACT SD 7.61). The workflow line is demonstrably usable on Astra, but these bundled workflow contrasts do not identify APP alone as the cause. Sol's reference comparison additionally carries the cohort limitation.

F-1.6.2 — Low APP mass accompanies weaker structure in both EXACT samples

The APP-priced invocation mechanism could encourage inlining, consistent with low mass and fewer functions. But the brief also differs in architecture and phase vocabulary; source inspection plus these means does not isolate causality. The explicit clarity guard is not a guarantee that output stays decomposed.

F-1.6.3 — Astra's floor concentrates the task into very few counted functions

The documented source inspection found domain logic inside nested callback chains in a small number of named functions. Correctness is perfect, but low LoC is not sufficient evidence of parsimony. Astra's native arms avoid this observed shape. The model comparison is date-confounded; generalization beyond this kata requires another controlled experiment.

F-1.6.4 — Astra isolation adds substantial time without a clear quality gain

Isolation costs 2.32× the observed time, with no better quality mean on these measures and perfect correctness in both arms. Sol shows 2.21× time on Claim Office, but that contrast is not contemporaneous. These observations do not establish that isolation is inherently harmful on every model or task.

F-1.6.5 — Astra correctness saturates across all four workflows

The Sol miss belongs to Isolated: one run passes 10/15 scenarios, yielding a cell mean near 93%. It exits normally and builds its CLI, so it is not an infrastructure exclusion. One miss does not reliably estimate a failure rate. Astra's all-green sample does not prove an absence of rare failures.

F-1.6.6 — Astra uses more observed time and tokens at every workflow

Token ratios range roughly 1.53–1.94×. Astra also has more marker-counted cycles in every structured arm (39.4–42.4 versus 28.0–33.4). More iterations may explain part of the gap, but model throughput cannot be separated from date, environment or test-list granularity here. This is a cohort association, not an isolated model-speed estimate.

F-1.6.7 — Astra uses fewer Production LoC across the observed workflows

Astra's Inline longest_function is 13.4 ± 2.79 versus Sol's 20.8 ± 2.39, but average function lengths (6.17 ± 1.59 versus 7.42 ± 1.04) overlap variation. The source is shorter without losing observed external correctness in these Inline cells. In Floor/EXACT, low LoC accompanies fewer, longer functions; it cannot be read as the same quality benefit. The Isolated correctness difference also prevents treating every row as identical completeness.

Caveats and scope

• One kata and one prompt style; no general result about small tasks or other routes.
• Five replicates and many outcomes: SD separation is descriptive, not a formal significance or equivalence test.
• Refactor markers include no-op reviews. Ratios to cycles reflect formatting and workflow contract, not verified improvements. For example, Sol Inline's means are 33.2/33.4, not an exact one-to-one identity.
• Pooled prediction accuracy (98.7–99.7% in structured cells) covers parsed blocks, not missing predictions. Already-green cycles need not produce failure blocks.
• cost_usd is not a declared outcome. The recomputed CSV costs use the lab's list-price table, not pi's inline costs or subscription invoices. Astra's undeclared model configuration makes inline pricing and context limits unsafe to interpret; transcript model identity is distinct from pricing metadata.
• Phase timing/context metrics are parser-limited on this route and remain outside the interpretation. The -no-thinking label does not establish reasoning off.

The practical preference is native Inline for structured Astra work in this sample, with the floor as the cheap but poorly decomposed alternative. Fresh same-image comparisons, and a brief-only intervention, are needed before turning cross-model or APP-mechanism associations into causal claims.