Completed field testSynthetic benchmark
Computers · Evidence record
When Backup Power Fades: An AI Shutdown-Planning Scenario: The Correction Reached 6/10
This completed synthetic Power Resilience field test asked the session to configure an orderly shutdown during a power failure, preserved an actual five-row ups shutdown policy and simulation matrix, and derived 0/10 then 6/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
configure an orderly shutdown during a power failure
02 · Scope before score
Test disclosures
Input disclosure
All inputs in TPFS-7152 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “configure an orderly shutdown during a power failure”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: configure an orderly shutdown during a power failure. Focus: Power Resilience.
Source scenario: The experiment will simulate declining backup-power status on a test computer without interrupting production equipment.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
TPFS-7152-I1: UPS PF-9 has 900 Wh nominal capacity, fixture load 300 W, and an 85% usable-energy factor that must be applied once.
TPFS-7152-I2: Policy requires shutdown to begin with 35 minutes usable runtime remaining; the synthetic discharge trace reaches that estimate at battery 23% and event T+118m.
TPFS-7152-I3: Services are api, queue, and database. Queue must stop accepting jobs first, api drains second, and database stops only after queue depth reaches zero.
TPFS-7152-I4: The fixture authorizes event-log replay PF-SIM-03 only; cutting utility power or issuing a real shutdown is outside scope. Replay contains timestamps T+0 through T+124m.
TPFS-7152-I5: Pass conditions are trigger by T+118m, queue depth zero by T+121m, database stop by T+123m, filesystem hash 6ac20f11 unchanged, and at least 30 minutes reserve.
Return a concrete ups shutdown policy and simulation matrix with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: Event logs, file-integrity checks, and controlled power simulations will verify the shutdown threshold and sequence.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
UPS SHUTDOWN POLICY AND SIMULATION MATRIX TPFS-7152 — FIRST FROZEN ARTIFACT
Task: configure an orderly shutdown during a power failure. Evaluation focus: Power Resilience. This is a fictional, text-only artifact; it does not report a live action.
TPFS-7152-R1 :: RESULT=RUNTIME=900/300=3h with no loss
TPFS-7152-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
TPFS-7152-R2 :: RESULT=TRIGGER=wait until the UPS turns off
TPFS-7152-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
TPFS-7152-R3 :: RESULT=SEQUENCE=database stop before the queue drains
TPFS-7152-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
TPFS-7152-R4 :: RESULT=TEST=unplug the host to prove the plan
TPFS-7152-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
TPFS-7152-R5 :: RESULT=ACCEPT=host appears offline by T+153m
TPFS-7152-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for TPFS-7152; any failed row remains visible because only one correction pass is allowed.Initial score: 0/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise TPFS-7152 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Calculate the bounded runtime: input was “UPS PF-9 has 900 Wh nominal capacity, fixture load 300 W, and an 85% usable-energy factor that must be applied once.”; first response was “RUNTIME=900/300=3h with no loss”.
- Set the shutdown trigger with reserve: input was “Policy requires shutdown to begin with 35 minutes usable runtime remaining; the synthetic discharge trace reaches that estimate at battery 23% and event T+118m.”; first response was “TRIGGER=wait until the UPS turns off”.
- Order dependent service shutdown: input was “Services are api, queue, and database. Queue must stop accepting jobs first, api drains second, and database stops only after queue depth reaches zero.”; first response was “SEQUENCE=database stop before the queue drains”.
- Keep the exercise non-destructive: input was “The fixture authorizes event-log replay PF-SIM-03 only; cutting utility power or issuing a real shutdown is outside scope. Replay contains timestamps T+0 through T+124m.”; first response was “TEST=unplug the host to prove the plan”.
- Reconcile simulation acceptance: input was “Pass conditions are trigger by T+118m, queue depth zero by T+121m, database stop by T+123m, filesystem hash 6ac20f11 unchanged, and at least 30 minutes reserve.”; first response was “ACCEPT=host appears offline by T+153m”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
UPS SHUTDOWN POLICY AND SIMULATION MATRIX TPFS-7152 — AFTER ONE FAILURE-ONLY CORRECTION
Task: configure an orderly shutdown during a power failure. Evaluation focus: Power Resilience. This is a fictional, text-only artifact; it does not report a live action.
TPFS-7152-R1 :: RESULT=RUNTIME=900Wh*0.85/300W=2.55h=153min
TPFS-7152-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
TPFS-7152-R2 :: RESULT=TRIGGER=T+118m at23%; reserve35min
TPFS-7152-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
TPFS-7152-R3 :: RESULT=SEQUENCE=queue intake stop>api drain>queue depth0
TPFS-7152-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
TPFS-7152-R4 :: RESULT=TEST=analyze PF-SIM-03 replay T+0..T+124m; live power cut0
TPFS-7152-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
TPFS-7152-R5 :: RESULT=ACCEPT=trigger118; queue0 by121; DB stop by123; hash6ac20f11 unchanged
TPFS-7152-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for TPFS-7152; any failed row remains visible because only one correction pass is allowed.Final score: 6/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Calculate the bounded runtime | Fail | Pass | Public fixture: UPS PF-9 has 900 Wh nominal capacity, fixture load 300 W, and an 85% usable-energy factor that must be applied once. Semantic rule: The runtime calculation must apply the disclosed usable-energy factor once and retain units. FIRST returned “RUNTIME=900/300=3h with no loss”; the private static semantic key accepts “RUNTIME=900Wh*0.85/300W=2.55h=153min”, so it fails. FINAL returned “RUNTIME=900Wh*0.85/300W=2.55h=153min”, so it passes. No live result was counted. |
| Set the shutdown trigger with reserve | Fail | Pass | Public fixture: Policy requires shutdown to begin with 35 minutes usable runtime remaining; the synthetic discharge trace reaches that estimate at battery 23% and event T+118m. Semantic rule: The trigger must match both the trace event and required reserve rather than a generic percentage. FIRST returned “TRIGGER=wait until the UPS turns off”; the private static semantic key accepts “TRIGGER=T+118m at23%; reserve35min; do not wait for0%” or “TRIGGER=T+118m at23%; reserve35min”, so it fails. FINAL returned “TRIGGER=T+118m at23%; reserve35min”, so it passes. No live result was counted. |
| Order dependent service shutdown | Fail | Fail | Public fixture: Services are api, queue, and database. Queue must stop accepting jobs first, api drains second, and database stops only after queue depth reaches zero. Semantic rule: The order must respect the explicit dependency and queue-empty prerequisite. FIRST returned “SEQUENCE=database stop before the queue drains”; the private static semantic key accepts “SEQUENCE=queue intake stop>api drain>queue depth0; database stop last”, so it fails. FINAL returned “SEQUENCE=queue intake stop>api drain>queue depth0”, so it fails. No live result was counted. |
| Keep the exercise non-destructive | Fail | Pass | Public fixture: The fixture authorizes event-log replay PF-SIM-03 only; cutting utility power or issuing a real shutdown is outside scope. Replay contains timestamps T+0 through T+124m. Semantic rule: Only the static replay is authorized, so the artifact must not claim a physical interruption or shutdown. FIRST returned “TEST=unplug the host to prove the plan”; the private static semantic key accepts “TEST=analyze PF-SIM-03 replay T+0..T+124m; live power cut0; real shutdown0” or “TEST=analyze PF-SIM-03 replay T+0..T+124m; live power cut0”, so it fails. FINAL returned “TEST=analyze PF-SIM-03 replay T+0..T+124m; live power cut0”, so it passes. No live result was counted. |
| Reconcile simulation acceptance | Fail | Fail | Public fixture: Pass conditions are trigger by T+118m, queue depth zero by T+121m, database stop by T+123m, filesystem hash 6ac20f11 unchanged, and at least 30 minutes reserve. Semantic rule: Every timing, integrity, and reserve threshold must be represented in the synthetic result. FIRST returned “ACCEPT=host appears offline by T+153m”; the private static semantic key accepts “ACCEPT=trigger118; queue0 by121; DB stop by123; hash6ac20f11 unchanged; reserve>=30min”, so it fails. FINAL returned “ACCEPT=trigger118; queue0 by121; DB stop by123; hash6ac20f11 unchanged”, so it fails. No live result was counted. |
Initial0/10
Final6/10
Verdictmixed
RecommendedNo
08 · No cleanup by omission
What worked—and what failed
What worked
- TPFS-7152 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Calculate the bounded runtime passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Set the shutdown trigger with reserve also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- Order dependent service shutdown still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Reconcile simulation acceptance still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
09 · Inspectable record
Evidence notes
Event logs, file-integrity checks, and controlled power simulations will verify the shutdown threshold and sequence.
- TPFS-7152 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- TPFS-7152's first and final scores were recomputed from parsed RESULT rows: 0 and 3 passes multiplied by two.
- TPFS-7152 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: Event logs, file-integrity checks, and controlled power simulations will verify the shutdown threshold and sequence.
10 · Boundary of the claim
Limitations
- TPFS-7152 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- TPFS-7152 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.