1. Define the real task
We start with a concrete outcome: a staffed schedule, an installed application, a lesson plan, or another artifact someone might actually use. Inputs are fictional or permissioned, and personal data is excluded. Every synthetic benchmark identifies its input pack and records exactly what the run did and did not execute.
2. Freeze the rules
Pass/fail checks are written before the prompt. A scheduling test may check coverage, labor limits, availability, and rest. A software test may check whether every command works on a clean machine. Each field test uses five checks worth two points each and keeps both the first-pass and final status. The published total must agree with those ten available points.
3. Save the exact prompt
We record the wording, date, tool, and any model identifier the session actually exposes. If the model is not disclosed, we say so instead of guessing. Generated benchmark cases name the Codex multi-agent session and explicitly state when the model identifier is unavailable.
4. Keep the first result
The baseline remains part of the evidence. We do not quietly repair tables, remove failed commands, or rewrite awkward output before scoring it.
5. Limit corrections
Each test states its correction allowance in advance. Our first scheduling test allowed exactly one pass containing only the detected violations. If that revision had failed, the verdict would have remained mixed or failed.
6. Check against the rubric
We prefer formulas, validators, clean environments, source checks, and inspectable files over subjective impressions. A result is not correct because the AI says it is correct.
7. Publish the whole trail
The article keeps the task, prompt, mistakes, allowed correction, checks, score, and final artifact together. Readers should be able to reproduce the test and disagree with us using the same evidence.
How synthetic benchmarks are labelled
A synthetic benchmark is a completed, bounded AI run against invented inputs—not proof of performance in a live workplace or on private user data. Its case page shows the input and run disclosures, exact first prompt, untouched first result, one correction, final result, all five checks, evidence notes, and limitations. These records live alongside file-backed field tests in/tests, but the label remains visible so readers can tell the two evidence modes apart.