MEASURED / DEVELOPMENT
Internal operational development-a · 2026-11-03
Prompt-stack comparison on one matched gateway route; production output filters and tools excluded.
Internal development dataset. This is a measured smoke/development report, not held-out evidence of superiority.
| Configuration | Task passes | 95% descriptive interval | Request failures | Latency p50 / p95 | Measured tariff cost |
|---|---|---|---|---|---|
| Standard baseline | 6 / 6100.0% | 61.0–100.0% | 0 | 14832 / 15529 ms | IDR 20.5601IDR 3.4267 / pass |
| Strong baseline | 6 / 6100.0% | 61.0–100.0% | 0 | 15419 / 17523 ms | IDR 12.3600IDR 2.0600 / pass |
| GT Lunar prompt candidate | 6 / 6100.0% | 61.0–100.0% | 0 | 14979 / 21879 ms | IDR 12.3354IDR 2.0559 / pass |
GT-only passes versus the strong baseline: 0. Strong-baseline-only passes: 0. Exact paired McNemar p = 1.0000. This exploratory comparison does not establish general superiority.
Dataset & acceptance
- Version
- ops-regression-development-a@1
- Coverage
- 6 unique cases · 1 repeat(s) · 0 exclusions
- Source
- Original Galactre internal operational regression tasks, authored with a coding assistant on 2026-11-03. Public source cases; not independently curated or an external benchmark. Holdout cases are reserved from prompt tuning, not independently reviewed.
- Permission
- Original internal acceptance examples; permitted for internal measurement and aggregate reporting.
- Grader
- gt-deterministic-v1 · exact string / structural JSON checks
Conditions & accounting
- Output limit
- 512 tokens per attempt
- Tools & retries
- none · 0 retries
- Sampling
- Provider defaults; temperature and seed are not overridden or guaranteed.
- Ordering
- Persisted shuffled order across arms; one request per case/repetition/arm.
- Cost basis
- Verified tokens valued at the frozen GT tariff; not a provider invoice or customer wallet debit.
- Tariff version
- galactre-commerce-v4
Prompt versions and evidence fingerprints
Report: evp_87c8058d20edbcf146d5f060f56595fc
Report SHA-256: 2473baacd0c0d33a81eec20c0f9899a2dd89b1bf03602edd2e0eef1252e42f34
Dataset SHA-256: ec84d77ca1b6703c30afe937b3d9ed37f569bd2c4f0baf58992d3fc169dc6269
Deployment SHA-256: 6a7e3bcc7ac2c98ea785c5dc6d8f43ebc1b01e09a0e3481e00de650d5b70086b
baseline: standard-baseline@1
SHA-256: 66ea7d804fc1a31be610722a6e422a100d642ec3850159c0054a309bc159ab4d
strong: strong-baseline@1
SHA-256: fe6cf54865c5bb3fb76d9ec90d418c054af1a79adbeab8e7c94b907765449323
gt: regression-gt-580a626593d75f54@1
SHA-256: f2c4bdca33a1ee3455712308cb0e9fd59332135ff23f2944fe0a70874c0a3e14
The download contains every attempt's status, grader, response hash, verified usage, timing and cost. Private prompts, reference answers, response text and routing configuration remain outside the public report.
Review and limitations
Deterministic results and accounting reviewed by a Galactre administrator; no independent peer review implied.
- Internal development dataset. This is a measured smoke/development report, not held-out evidence of superiority.
- Prompt-stack measurement excludes production output filters, tools and agent orchestration. It does not measure the full production chat experience.
- Deterministic exact/JSON grading measures defined task acceptance only. No calibrated human quality review or general reasoning benchmark is implied.
- Model revision, sampling seed and temperature are not guaranteed by the provider. The fingerprint compares model labels and route names; endpoint revisions and silent weight updates are not exposed.
- Wilson intervals are descriptive at attempt level. Repeated cases are correlated; McNemar comparisons are exploratory, without multiplicity correction.
- Latency includes authorization, encrypted input upload, preflight, generation and input cleanup; single-worker sequential requests. Cache and time-of-run effects may differ.
- All planned attempts enter the denominator. Failed preflight requests count as failures. Unknown execution costs block publication.
- No universal advantage or independently trained model weights can be inferred from this report.

Galactre Studio