Independent Testing Methodology
The protocol used when this site publishes its own GPT-6 Astra test results.
Status: Methodology published. Independent benchmark results are only added after reproducible runs are completed.
Principles
- Record the exact model ID or snapshot and test date.
- Keep prompts and scoring criteria stable across compared models.
- Run multiple trials where randomness or latency matters.
- Separate correctness, speed and cost instead of collapsing them into one score.
- Publish failures and ambiguous cases, not only successful examples.
Planned test groups
| Test | Primary metric | Notes |
|---|---|---|
| Python coding | Pass rate | Unit-test based tasks |
| SQL | Correctness | Schema-grounded queries |
| Structured output | Schema validity | Repeated JSON-schema trials |
| Long context | Retrieval accuracy | Controlled needle placement |
| Latency | Median / p95 | Same region and request shape |
| Tool use | Task completion | Multi-step tool workflows |
Reproducibility record
Every published test should include prompt set version, runtime settings, reasoning effort, input/output token counts when available, and any scoring code needed to reproduce the result.