MEASURED / DEVELOPMENT
Internal operational development-b · 2026-11-03
Prompt-stack comparison on one matched gateway route; production output filters and tools excluded.
Internal development dataset. This is a measured smoke/development report, not held-out evidence of superiority.
| Configuration | Task passes | 95% descriptive interval | Request failures | Latency p50 / p95 | Measured tariff cost |
|---|---|---|---|---|---|
| Standard baseline | 6 / 6100.0% | 61.0–100.0% | 0 | 14487 / 15923 ms | IDR 9.2831IDR 1.5472 / pass |
| Strong baseline | 6 / 6100.0% | 61.0–100.0% | 0 | 15464 / 16500 ms | IDR 10.0926IDR 1.6821 / pass |
| GT Lunar prompt candidate | 6 / 6100.0% | 61.0–100.0% | 0 | 14717 / 20417 ms | IDR 10.9645IDR 1.8274 / pass |
GT-only passes versus the strong baseline: 0. Strong-baseline-only passes: 0. Exact paired McNemar p = 1.0000. This exploratory comparison does not establish general superiority.
Dataset & acceptance
- Version
- ops-regression-development-b@1
- Coverage
- 6 unique cases · 1 repeat(s) · 0 exclusions
- Source
- Original Galactre internal operational regression tasks, authored with a coding assistant on 2026-11-03. Public source cases; not independently curated or an external benchmark. Holdout cases are reserved from prompt tuning, not independently reviewed.
- Permission
- Original internal acceptance examples; permitted for internal measurement and aggregate reporting.
- Grader
- gt-deterministic-v1 · exact string / structural JSON checks
Conditions & accounting
- Output limit
- 512 tokens per attempt
- Tools & retries
- none · 0 retries
- Sampling
- Provider defaults; temperature and seed are not overridden or guaranteed.
- Ordering
- Persisted shuffled order across arms; one request per case/repetition/arm.
- Cost basis
- Verified tokens valued at the frozen GT tariff; not a provider invoice or customer wallet debit.
- Tariff version
- galactre-commerce-v4
Prompt versions and evidence fingerprints
Report: evp_656a633b1fc56ec83b7e4f25ef50bc84
Report SHA-256: a58fc65185d29c19d6ac246e4d2a0b3fc80b5088019788ded703296d6dcb8026
Dataset SHA-256: 999cf298a93b9116415d1cfe7afe899701cb5e3c61bdf59a578fa98a5708418d
Deployment SHA-256: 6a7e3bcc7ac2c98ea785c5dc6d8f43ebc1b01e09a0e3481e00de650d5b70086b
baseline: standard-baseline@1
SHA-256: 66ea7d804fc1a31be610722a6e422a100d642ec3850159c0054a309bc159ab4d
strong: strong-baseline@1
SHA-256: fe6cf54865c5bb3fb76d9ec90d418c054af1a79adbeab8e7c94b907765449323
gt: regression-gt-580a626593d75f54@1
SHA-256: f2c4bdca33a1ee3455712308cb0e9fd59332135ff23f2944fe0a70874c0a3e14
The download contains every attempt's status, grader, response hash, verified usage, timing and cost. Private prompts, reference answers, response text and routing configuration remain outside the public report.
Review and limitations
Deterministic results and accounting reviewed by a Galactre administrator; no independent peer review implied.
- Internal development dataset. This is a measured smoke/development report, not held-out evidence of superiority.
- Prompt-stack measurement excludes production output filters, tools and agent orchestration. It does not measure the full production chat experience.
- Deterministic exact/JSON grading measures defined task acceptance only. No calibrated human quality review or general reasoning benchmark is implied.
- Model revision, sampling seed and temperature are not guaranteed by the provider. The fingerprint compares model labels and route names; endpoint revisions and silent weight updates are not exposed.
- Wilson intervals are descriptive at attempt level. Repeated cases are correlated; McNemar comparisons are exploratory, without multiplicity correction.
- Latency includes authorization, encrypted input upload, preflight, generation and input cleanup; single-worker sequential requests. Cache and time-of-run effects may differ.
- All planned attempts enter the denominator. Failed preflight requests count as failures. Unknown execution costs block publication.
- No universal advantage or independently trained model weights can be inferred from this report.

Galactre Studio