Completed field testSynthetic benchmark
Computers · Evidence record
A Slow SQL Query and an AI Triage Protocol: All Five Semantic Checks Passed
This completed synthetic Query Performance field test asked the session to triage a slow SQL query using a supplied execution plan, preserved an actual five-row software patch and test record, and derived 6/10 then 10/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
triage a slow SQL query using a supplied execution plan
02 · Scope before score
Test disclosures
Input disclosure
All inputs in TSSQ-4234 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “triage a slow SQL query using a supplied execution plan”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: triage a slow SQL query using a supplied execution plan. Focus: Query Performance.
Source scenario: The experiment will provide a synthetic schema, data distribution, query, execution plan, indexes, and a controlled performance baseline.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
TSSQ-4234-I1: Plan node Seq Scan estimates 1,200 rows but returns 184,000; filter is tenant_id=42 and created_at>=2026-01-01.
TSSQ-4234-I2: Existing index is (tenant_id); candidate is (tenant_id, created_at) INCLUDE (status).
TSSQ-4234-I3: Expected result count is 184,000 and checksum is 9ab41c70.
TSSQ-4234-I4: Baseline median is 4.8 s over 5 runs; target is below 900 ms median over 5 clean-cache runs.
TSSQ-4234-I5: Candidate index adds 7% fixture write time; allowed ceiling is 10%.
Return a concrete software patch and test record with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: Benchmark runs on a disposable database will verify the bottleneck, semantic equivalence, latency, resource use, and regression risk.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
SOFTWARE PATCH AND TEST RECORD TSSQ-4234 — FIRST FROZEN ARTIFACT
Task: triage a slow SQL query using a supplied execution plan. Evaluation focus: Query Performance. This is a fictional, text-only artifact; it does not report a live action.
TSSQ-4234-R1 :: RESULT=PLAN=Seq Scan estimate1200 actual184000; cardinality underestimate
TSSQ-4234-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
TSSQ-4234-R2 :: RESULT=INDEX=(tenant_id,created_at) INCLUDE(status)
TSSQ-4234-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
TSSQ-4234-R3 :: RESULT=SEMANTICS=184000 rows hash9ab41c70 before and after
TSSQ-4234-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
TSSQ-4234-R4 :: RESULT=MEASURE=one warm-cache run
TSSQ-4234-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
TSSQ-4234-R5 :: RESULT=TRADEOFF=no write cost
TSSQ-4234-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for TSSQ-4234; any failed row remains visible because only one correction pass is allowed.Initial score: 6/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise TSSQ-4234 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Use bounded measurement: input was “Baseline median is 4.8 s over 5 runs; target is below 900 ms median over 5 clean-cache runs.”; first response was “MEASURE=one warm-cache run”.
- State residual write cost: input was “Candidate index adds 7% fixture write time; allowed ceiling is 10%.”; first response was “TRADEOFF=no write cost”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
SOFTWARE PATCH AND TEST RECORD TSSQ-4234 — AFTER ONE FAILURE-ONLY CORRECTION
Task: triage a slow SQL query using a supplied execution plan. Evaluation focus: Query Performance. This is a fictional, text-only artifact; it does not report a live action.
TSSQ-4234-R1 :: RESULT=PLAN=Seq Scan estimate1200 actual184000; cardinality underestimate
TSSQ-4234-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
TSSQ-4234-R2 :: RESULT=INDEX=(tenant_id,created_at) INCLUDE(status)
TSSQ-4234-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
TSSQ-4234-R3 :: RESULT=SEMANTICS=184000 rows hash9ab41c70 before and after
TSSQ-4234-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
TSSQ-4234-R4 :: RESULT=MEASURE=5 baseline median4.8s; 5 clean-cache target<900ms
TSSQ-4234-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
TSSQ-4234-R5 :: RESULT=TRADEOFF=7% write cost within 10% ceiling
TSSQ-4234-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for TSSQ-4234; any failed row remains visible because only one correction pass is allowed.Final score: 10/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Read the supplied plan cardinality | Pass | Pass | Public fixture: Plan node Seq Scan estimates 1,200 rows but returns 184,000; filter is tenant_id=42 and created_at>=2026-01-01. Semantic rule: The estimate/actual mismatch is the primary plan anomaly. FIRST returned “PLAN=Seq Scan estimate1200 actual184000; cardinality underestimate”; the private static semantic key accepts “PLAN=Seq Scan estimate1200 actual184000; cardinality underestimate”, so it passes. FINAL returned “PLAN=Seq Scan estimate1200 actual184000; cardinality underestimate”, so it passes. No live result was counted. |
| Use the available index definition | Pass | Pass | Public fixture: Existing index is (tenant_id); candidate is (tenant_id, created_at) INCLUDE (status). Semantic rule: The candidate matches equality then range predicates and covers status. FIRST returned “INDEX=(tenant_id,created_at) INCLUDE(status)”; the private static semantic key accepts “INDEX=(tenant_id,created_at) INCLUDE(status)”, so it passes. FINAL returned “INDEX=(tenant_id,created_at) INCLUDE(status)”, so it passes. No live result was counted. |
| Preserve query semantics | Pass | Pass | Public fixture: Expected result count is 184,000 and checksum is 9ab41c70. Semantic rule: Performance work cannot change the result count or frozen checksum. FIRST returned “SEMANTICS=184000 rows hash9ab41c70 before and after”; the private static semantic key accepts “SEMANTICS=184000 rows hash9ab41c70 before and after”, so it passes. FINAL returned “SEMANTICS=184000 rows hash9ab41c70 before and after”, so it passes. No live result was counted. |
| Use bounded measurement | Fail | Pass | Public fixture: Baseline median is 4.8 s over 5 runs; target is below 900 ms median over 5 clean-cache runs. Semantic rule: Both sample count, cache condition, and threshold are fixed. FIRST returned “MEASURE=one warm-cache run”; the private static semantic key accepts “MEASURE=5 baseline median4.8s; 5 clean-cache target<900ms”, so it fails. FINAL returned “MEASURE=5 baseline median4.8s; 5 clean-cache target<900ms”, so it passes. No live result was counted. |
| State residual write cost | Fail | Pass | Public fixture: Candidate index adds 7% fixture write time; allowed ceiling is 10%. Semantic rule: The measured write overhead must be reported against the ceiling. FIRST returned “TRADEOFF=no write cost”; the private static semantic key accepts “TRADEOFF=7% write cost within 10% ceiling”, so it fails. FINAL returned “TRADEOFF=7% write cost within 10% ceiling”, so it passes. No live result was counted. |
Initial6/10
Final10/10
Verdictworked
RecommendedYes, for this scope
08 · No cleanup by omission
What worked—and what failed
What worked
- TSSQ-4234 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Read the supplied plan cardinality passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Use the available index definition also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- The first artifact failed Use bounded measurement; the one permitted correction resolved it, but the initial defect remains published.
09 · Inspectable record
Evidence notes
Benchmark runs on a disposable database will verify the bottleneck, semantic equivalence, latency, resource use, and regression risk.
- TSSQ-4234 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- TSSQ-4234's first and final scores were recomputed from parsed RESULT rows: 3 and 5 passes multiplied by two.
- TSSQ-4234 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: Benchmark runs on a disposable database will verify the bottleneck, semantic equivalence, latency, resource use, and regression risk.
10 · Boundary of the claim
Limitations
- TSSQ-4234 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- TSSQ-4234 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.