MEASURED / HOLDOUT
Internal operational holdout-b · 2026-11-03
Prompt-stack comparison on one matched gateway route; production output filters and tools excluded.
Holdout is a dataset split label, not independent review. These original internal, public-source cases were reserved from prompt tuning. Results describe only this task set and frozen prompt versions; do not tune against them and reuse them as untouched evidence.
| Configuration | Task passes | 95% descriptive interval | Request failures | Latency p50 / p95 | Measured tariff cost |
|---|---|---|---|---|---|
| Standard baseline | 6 / 6100.0% | 61.0–100.0% | 0 | 10787 / 12414 ms | IDR 18.6675IDR 3.1113 / pass |
| Strong baseline | 6 / 6100.0% | 61.0–100.0% | 0 | 14695 / 16327 ms | IDR 11.6638IDR 1.9440 / pass |
| GT Lunar prompt candidate | 6 / 6100.0% | 61.0–100.0% | 0 | 11439 / 15192 ms | IDR 10.6161IDR 1.7694 / pass |
GT-only passes versus the strong baseline: 0. Strong-baseline-only passes: 0. Exact paired McNemar p = 1.0000. This exploratory comparison does not establish general superiority.
Dataset & acceptance
- Version
- ops-regression-holdout-b@1
- Coverage
- 6 unique cases · 1 repeat(s) · 0 exclusions
- Source
- Original Galactre internal operational regression tasks, authored with a coding assistant on 2026-11-03. Public source cases; not independently curated or an external benchmark. Holdout cases are reserved from prompt tuning, not independently reviewed.
- Permission
- Original internal acceptance examples; permitted for internal measurement and aggregate reporting.
- Grader
- gt-deterministic-v1 · exact string / structural JSON checks
Conditions & accounting
- Output limit
- 512 tokens per attempt
- Tools & retries
- none · 0 retries
- Sampling
- Provider defaults; temperature and seed are not overridden or guaranteed.
- Ordering
- Persisted shuffled order across arms; one request per case/repetition/arm.
- Cost basis
- Verified tokens valued at the frozen GT tariff; not a provider invoice or customer wallet debit.
- Tariff version
- galactre-commerce-v4
Prompt versions and evidence fingerprints
Report: evp_1f451782456395f6d949ae3b3fc05b54
Report SHA-256: 1942522bad52108754485aaa1755cc9e75e1219257fefb7cc854f21a063cdbbe
Dataset SHA-256: a65c15d9e99afcca0a6f45f66596fa6ea3f7e1f25c240eb4c458be700ff14610
Deployment SHA-256: 6a7e3bcc7ac2c98ea785c5dc6d8f43ebc1b01e09a0e3481e00de650d5b70086b
baseline: standard-baseline@1
SHA-256: 66ea7d804fc1a31be610722a6e422a100d642ec3850159c0054a309bc159ab4d
strong: strong-baseline@1
SHA-256: fe6cf54865c5bb3fb76d9ec90d418c054af1a79adbeab8e7c94b907765449323
gt: regression-gt-580a626593d75f54@1
SHA-256: f2c4bdca33a1ee3455712308cb0e9fd59332135ff23f2944fe0a70874c0a3e14
The download contains every attempt's status, grader, response hash, verified usage, timing and cost. Private prompts, reference answers, response text and routing configuration remain outside the public report.
Review and limitations
Deterministic results and accounting reviewed by a Galactre administrator; no independent peer review implied.
- Holdout is a dataset split label, not independent review. These original internal, public-source cases were reserved from prompt tuning. Results describe only this task set and frozen prompt versions; do not tune against them and reuse them as untouched evidence.
- Prompt-stack measurement excludes production output filters, tools and agent orchestration. It does not measure the full production chat experience.
- Deterministic exact/JSON grading measures defined task acceptance only. No calibrated human quality review or general reasoning benchmark is implied.
- Model revision, sampling seed and temperature are not guaranteed by the provider. The fingerprint compares model labels and route names; endpoint revisions and silent weight updates are not exposed.
- Wilson intervals are descriptive at attempt level. Repeated cases are correlated; McNemar comparisons are exploratory, without multiplicity correction.
- Latency includes authorization, encrypted input upload, preflight, generation and input cleanup; single-worker sequential requests. Cache and time-of-run effects may differ.
- All planned attempts enter the denominator. Failed preflight requests count as failures. Unknown execution costs block publication.
- No universal advantage or independently trained model weights can be inferred from this report.

Galactre Studio