MEASURED / HOLDOUT
Internal operational holdout-a · 2026-11-03
Prompt-stack comparison on one matched gateway route; production output filters and tools excluded.
Holdout is a dataset split label, not independent review. These original internal, public-source cases were reserved from prompt tuning. Results describe only this task set and frozen prompt versions; do not tune against them and reuse them as untouched evidence.
| Configuration | Task passes | 95% descriptive interval | Request failures | Latency p50 / p95 | Measured tariff cost |
|---|---|---|---|---|---|
| Standard baseline | 6 / 6100.0% | 61.0–100.0% | 0 | 12186 / 14687 ms | IDR 9.8721IDR 1.6454 / pass |
| Strong baseline | 6 / 6100.0% | 61.0–100.0% | 0 | 16898 / 28206 ms | IDR 10.7604IDR 1.7934 / pass |
| GT Lunar prompt candidate | 6 / 6100.0% | 61.0–100.0% | 0 | 17015 / 23793 ms | IDR 12.7361IDR 2.1227 / pass |
GT-only passes versus the strong baseline: 0. Strong-baseline-only passes: 0. Exact paired McNemar p = 1.0000. This exploratory comparison does not establish general superiority.
Dataset & acceptance
- Version
- ops-regression-holdout-a@1
- Coverage
- 6 unique cases · 1 repeat(s) · 0 exclusions
- Source
- Original Galactre internal operational regression tasks, authored with a coding assistant on 2026-11-03. Public source cases; not independently curated or an external benchmark. Holdout cases are reserved from prompt tuning, not independently reviewed.
- Permission
- Original internal acceptance examples; permitted for internal measurement and aggregate reporting.
- Grader
- gt-deterministic-v1 · exact string / structural JSON checks
Conditions & accounting
- Output limit
- 512 tokens per attempt
- Tools & retries
- none · 0 retries
- Sampling
- Provider defaults; temperature and seed are not overridden or guaranteed.
- Ordering
- Persisted shuffled order across arms; one request per case/repetition/arm.
- Cost basis
- Verified tokens valued at the frozen GT tariff; not a provider invoice or customer wallet debit.
- Tariff version
- galactre-commerce-v4
Prompt versions and evidence fingerprints
Report: evp_966ae55c2f4e5887a8c0704c113db307
Report SHA-256: b42bd91152695b2979d7ec4d8f33f0ecdf6b084b8355fd7d08194e6940cc34db
Dataset SHA-256: 2b6205c4de8190f8fff39a7e8f5b9d4f052f2ac87a1d2940230a8a2367db47cd
Deployment SHA-256: 6a7e3bcc7ac2c98ea785c5dc6d8f43ebc1b01e09a0e3481e00de650d5b70086b
baseline: standard-baseline@1
SHA-256: 66ea7d804fc1a31be610722a6e422a100d642ec3850159c0054a309bc159ab4d
strong: strong-baseline@1
SHA-256: fe6cf54865c5bb3fb76d9ec90d418c054af1a79adbeab8e7c94b907765449323
gt: regression-gt-580a626593d75f54@1
SHA-256: f2c4bdca33a1ee3455712308cb0e9fd59332135ff23f2944fe0a70874c0a3e14
The download contains every attempt's status, grader, response hash, verified usage, timing and cost. Private prompts, reference answers, response text and routing configuration remain outside the public report.
Review and limitations
Deterministic results and accounting reviewed by a Galactre administrator; no independent peer review implied.
- Holdout is a dataset split label, not independent review. These original internal, public-source cases were reserved from prompt tuning. Results describe only this task set and frozen prompt versions; do not tune against them and reuse them as untouched evidence.
- Prompt-stack measurement excludes production output filters, tools and agent orchestration. It does not measure the full production chat experience.
- Deterministic exact/JSON grading measures defined task acceptance only. No calibrated human quality review or general reasoning benchmark is implied.
- Model revision, sampling seed and temperature are not guaranteed by the provider. The fingerprint compares model labels and route names; endpoint revisions and silent weight updates are not exposed.
- Wilson intervals are descriptive at attempt level. Repeated cases are correlated; McNemar comparisons are exploratory, without multiplicity correction.
- Latency includes authorization, encrypted input upload, preflight, generation and input cleanup; single-worker sequential requests. Cache and time-of-run effects may differ.
- All planned attempts enter the denominator. Failed preflight requests count as failures. Unknown execution costs block publication.
- No universal advantage or independently trained model weights can be inferred from this report.

Galactre Studio