Galactre
USD

Choose how catalog prices and estimates appear. Checkout and account records remain in IDR.

Open Chat

GALACTRE / MODEL EVALUATIONS

Evidence before
rankings.

A useful answer needs more than a confident voice. Explore how to evaluate Galactre for your work, what to measure, and the evidence a meaningful comparison needs.

01 / THE CURRENT PICTURE

A clear view of what
is available and measured.

Check service availability alongside the publication status for each family. A model being available does not imply a benchmark score or a position in a ranking.

Model availability could not be refreshed. Evaluation publication status remains separate.

Measured reports

Versioned evidence · no automatic ranking

MEASURED / HOLDOUT

Internal operational holdout-a · 2026-11-03

Prompt-stack comparison on one matched gateway route; production output filters and tools excluded.

Holdout is a dataset split label, not independent review. These original internal, public-source cases were reserved from prompt tuning. Results describe only this task set and frozen prompt versions; do not tune against them and reuse them as untouched evidence.

All planned attempts are included. Costs are verified token consumption valued at the recorded GT tariff.
ConfigurationTask passes95% descriptive intervalRequest failuresLatency p50 / p95Measured tariff cost
Standard baseline6 / 6100.0%61.0–100.0%012186 / 14687 msIDR 9.8721IDR 1.6454 / pass
Strong baseline6 / 6100.0%61.0–100.0%016898 / 28206 msIDR 10.7604IDR 1.7934 / pass
GT Lunar prompt candidate6 / 6100.0%61.0–100.0%017015 / 23793 msIDR 12.7361IDR 2.1227 / pass

GT-only passes versus the strong baseline: 0. Strong-baseline-only passes: 0. Exact paired McNemar p = 1.0000. This exploratory comparison does not establish general superiority.

Dataset & acceptance

Version
ops-regression-holdout-a@1
Coverage
6 unique cases · 1 repeat(s) · 0 exclusions
Source
Original Galactre internal operational regression tasks, authored with a coding assistant on 2026-11-03. Public source cases; not independently curated or an external benchmark. Holdout cases are reserved from prompt tuning, not independently reviewed.
Permission
Original internal acceptance examples; permitted for internal measurement and aggregate reporting.
Grader
gt-deterministic-v1 · exact string / structural JSON checks

Conditions & accounting

Output limit
512 tokens per attempt
Tools & retries
none · 0 retries
Sampling
Provider defaults; temperature and seed are not overridden or guaranteed.
Ordering
Persisted shuffled order across arms; one request per case/repetition/arm.
Cost basis
Verified tokens valued at the frozen GT tariff; not a provider invoice or customer wallet debit.
Tariff version
galactre-commerce-v4
Prompt versions and evidence fingerprints
Report: evp_966ae55c2f4e5887a8c0704c113db307
Report SHA-256: b42bd91152695b2979d7ec4d8f33f0ecdf6b084b8355fd7d08194e6940cc34db
Dataset SHA-256: 2b6205c4de8190f8fff39a7e8f5b9d4f052f2ac87a1d2940230a8a2367db47cd
Deployment SHA-256: 6a7e3bcc7ac2c98ea785c5dc6d8f43ebc1b01e09a0e3481e00de650d5b70086b
baseline: standard-baseline@1
SHA-256: 66ea7d804fc1a31be610722a6e422a100d642ec3850159c0054a309bc159ab4d
strong: strong-baseline@1
SHA-256: fe6cf54865c5bb3fb76d9ec90d418c054af1a79adbeab8e7c94b907765449323
gt: regression-gt-580a626593d75f54@1
SHA-256: f2c4bdca33a1ee3455712308cb0e9fd59332135ff23f2944fe0a70874c0a3e14

The download contains every attempt's status, grader, response hash, verified usage, timing and cost. Private prompts, reference answers, response text and routing configuration remain outside the public report.

Review and limitations

Deterministic results and accounting reviewed by a Galactre administrator; no independent peer review implied.

  • Holdout is a dataset split label, not independent review. These original internal, public-source cases were reserved from prompt tuning. Results describe only this task set and frozen prompt versions; do not tune against them and reuse them as untouched evidence.
  • Prompt-stack measurement excludes production output filters, tools and agent orchestration. It does not measure the full production chat experience.
  • Deterministic exact/JSON grading measures defined task acceptance only. No calibrated human quality review or general reasoning benchmark is implied.
  • Model revision, sampling seed and temperature are not guaranteed by the provider. The fingerprint compares model labels and route names; endpoint revisions and silent weight updates are not exposed.
  • Wilson intervals are descriptive at attempt level. Repeated cases are correlated; McNemar comparisons are exploratory, without multiplicity correction.
  • Latency includes authorization, encrypted input upload, preflight, generation and input cleanup; single-worker sequential requests. Cache and time-of-run effects may differ.
  • All planned attempts enter the denominator. Failed preflight requests count as failures. Unknown execution costs block publication.
  • No universal advantage or independently trained model weights can be inferred from this report.
Download measured JSON Published 2026-11-02T17:36:45+00:00
Model availability and reviewed evaluation publication status, without relative ranking.
ModelService statusEvaluation evidenceExplore
GT Lunargt-lunarStatus unavailablePlannedNot publishedNo scored report to compareModel profile
GT Stellargt-stellarStatus unavailablePlannedNot publishedNo scored report to compareModel profile
GT Nebulargt-nebularStatus unavailablePlannedNot publishedNo scored report to compareModel profile
GT Cosmicgt-cosmicStatus unavailablePlannedNot publishedNo scored report to compareModel profile

Scroll the comparison to see all columns.

The rows follow the model family order, not a quality ranking. Unknown results are not zero scores.

02 / DESIGN THE EVALUATION

Start with the work.
Keep the test repeatable.

This practical guide is a starting point for your own evaluation. It describes the evidence to collect, rather than a completed Galactre benchmark campaign.

Prepare a report
  1. 01

    Define the task

    Start with the decision or workflow you need to support. Define a useful answer, unacceptable failures and the population of inputs before choosing a model.

    Write acceptance criteria before looking at responses. Keep quality, operational limits and human review requirements separate.

  2. 02

    Build a representative set

    Include ordinary requests, ambiguous instructions, edge cases and deliberately incomplete information. Keep a held-out set for the final comparison.

    Version the dataset, document its origin and permissions, and remove personal data and credentials. A few examples are a smoke test, not a population estimate.

  3. 03

    Hold conditions steady

    Record the public model ID, date, exact prompt, supplied context and any settings exposed by your integration. Use the same conditions for each candidate.

    Repeat cases to observe variation. Record unavailable settings as unknown rather than assuming defaults or an identical model revision.

  4. 04

    Review and reproduce

    Apply the predefined rubric, inspect failures and retain enough evidence for another reviewer to understand the result. Separate a generated answer from an executed test.

    Publish denominators, exclusions, uncertainty and limitations with the result. Retest when the model release, prompt, dataset or application changes.

03 / UNDERSTAND THE SIGNALS

One number rarely
tells the whole story.

Keep response quality, reliability and operational efficiency visible separately. Choose measures that map to the outcome your application needs.

Judge whether the response completes the task correctly, using the reference material and constraints you supplied. Fluency alone is not evidence of correctness.

Task success

The share of valid cases that meet a predefined acceptance rule.

How to measure
Passing cases / valid evaluated cases
What to record
Publish the rubric, number of cases, exclusions and how ties or partially correct answers were handled.

Do not compare percentages produced from different datasets as if they were equivalent.

Grounding and instruction following

Whether claims are supported by the supplied material and explicit constraints are respected.

How to measure
Review facts, citations, format and required constraints separately
What to record
Keep the source, expected behavior and reviewer decision with each case. Mark missing information rather than rewarding invented details.

A confident answer can still introduce facts that were never in the input.

04 / A SMALL TEST LAB

Try a case.
Inspect the reasoning.

These illustrative prompts pair a task with review criteria. They are a starting point for manual testing, not a validated benchmark dataset. Opening a prompt prepares a draft in chat; you decide when to send it.

Preserve facts. Respect the brief.

A small case for checking factual preservation, length and tone. The supplied release note is the only factual source.

What to check

  • The rewritten release note contains at most 80 words.
  • The retry limit and activity-log behavior remain accurate.
  • Missing delay and release-date information is identified without invented values.
  • Any additional claims are traceable to the supplied source.
Open this prompt in chat
ILLUSTRATIVE TEST / Grounded writing
Rewrite the release note below for developers in no more than 80 words. Preserve all facts. Do not invent availability dates or platform support. Then list any missing information separately.

Source: The PHP job runner now retries failed jobs up to three times. Administrators can inspect each attempt in the activity log. Retry delays and the release date have not been specified.

05 / MAKE THE EVIDENCE USEFUL

A result someone else
can understand.

Use this planning checklist to prepare an evaluation. Checking an item records your local planning progress; it does not validate evidence or publish a result.

Evaluation planning checklist

0 of 6 checked
Mark the planning items you have prepared

Planning progress stays on this page and resets when it is reloaded. No data is sent to a server.

BLANK REPORT / JSON
{
    "schema": "galactre.evaluation-report.example.v1",
    "status": "draft",
    "model": {
        "id": "gt-lunar",
        "release": null
    },
    "evaluated_at": null,
    "task": {
        "name": null,
        "acceptance_rule": null
    },
    "dataset": {
        "version": null,
        "sample_count": null,
        "provenance": null,
        "exclusions": []
    },
    "conditions": {
        "prompt_version": null,
        "context": null,
        "known_settings": null,
        "repetitions": null
    },
    "results": {
        "passed": null,
        "failed": null,
        "request_errors": null,
        "latency_p50_ms": null,
        "latency_p95_ms": null,
        "verified_cost": null,
        "currency": null
    },
    "evidence": [],
    "review": {
        "reviewers": [],
        "limitations": []
    }
}

Example report structure, not an ingestion API. Null fields mean unrecorded values.

06 / READ RESULTS IN CONTEXT

Know what a result
can actually tell you.

Good reporting explains its boundaries as carefully as its findings. Apply these checks to your own tests and to comparisons you encounter elsewhere.

A sample is not a benchmark

The example cases on this page help you design a review. They do not form a validated suite, represent all users or establish a public model ranking.

Compare like with like

Different prompts, languages, context lengths, graders or test dates can change outcomes. Report these differences before drawing conclusions from a score.

Keep human review where it matters

Automated checks and model-based grading can miss errors or introduce bias. Validate a grading method against human-reviewed cases and inspect disagreements.

Revisit results as the product changes

A result describes the tested configuration at a point in time. Availability, prompts, operational conditions and preview behavior can change. Retain the original report when repeating a test.

07 / QUESTIONS, ANSWERED

Before you compare.

Does an evaluation cover every version in a GT family?

No. Read the model ID and measurement scope in the published evidence. New campaigns can pin a published GT release, its manifest hash and its sealed prompt. Earlier unversioned reports remain unversioned; a default promotion does not transfer their results to a new release.

Where is the ranked leaderboard?

Reviewed measured reports appear in the results section when published. The model status section shows service availability separately. A ranking needs a documented test set, comparable conditions and reviewable results; availability alone cannot establish quality.

Is this an official Galactre benchmark methodology?

This page provides a practical evaluation guide and a blank reporting template. It is not a claim that Galactre has completed a benchmark campaign or validated the example cases as a standard benchmark suite. A published result must identify its actual methodology.

Can I run the example tests from this page?

Open an example in chat to prepare a draft, review it and send it yourself. The page does not run models or spend credits automatically. A chat answer still needs independent review against the listed criteria.

Why are Stellar and Nebular listed without scores?

They are members of the GT model family. Availability is read from the current catalog, while evaluation evidence is separate. A planned or unavailable model is not represented as tested, and a credit purchase does not activate it.

Can I compare GT with other AI models?

Use the same authorized dataset, acceptance rules and recorded conditions for every candidate. Keep quality, latency, reliability and cost visible separately. Do not infer a head-to-head win from unrelated public scores or different model revisions.

Does downloading the template publish my results?

No. Download and copy operate in your browser. The template contains blank result fields; the planning checklist is local and is not saved to your account. This page does not upload answers, create an evaluation run or publish a report.

What should I record when settings or prices are unknown?

Use an explicit unknown value and state the limitation. Do not invent a model revision, sampling setting, price or token count. Report the values actually exposed by your integration and retain the evidence used for any calculated cost.

FROM EXPLORATION TO EVIDENCE

Bring a real task.
Define a useful result.

Start with an available model, keep the scope focused and review the answer against the work you need to do.