Completed field testSynthetic benchmark
Computers · Evidence record
An AI Checklist for a Low-Risk Firmware Update: Three Semantic Checks Still Failed
This completed synthetic Firmware field test asked the session to plan a low-risk firmware update, preserved an actual five-row firmware update safety checklist, and derived 0/10 then 4/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
plan a low-risk firmware update
02 · Scope before score
Test disclosures
Input disclosure
All inputs in PFU-9071 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “plan a low-risk firmware update”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: plan a low-risk firmware update. Focus: Firmware.
Source scenario: The experiment will ask for an update checklist based on a device model, current version, power conditions, and recovery options.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
PFU-9071-I1: Device board ID MB-7A2 is on firmware 1.14; signed package FW-MB7A2-1.18 targets MB-7A2 only.
PFU-9071-I2: Vendor rule requires AC power and battery at least 50%; fixture shows AC connected and battery 63%.
PFU-9071-I3: Baseline export CFG-114 has SHA-256 31bd10ae; Secure Boot is on and virtualization is enabled.
PFU-9071-I4: Recovery requires USB REC-MB7A2, rear port U2, and recovery button held for 5 seconds with power off.
PFU-9071-I5: Static acceptance is version 1.18 reported twice, Secure Boot on, virtualization enabled, and three cold boots with zero firmware errors.
Return a concrete firmware update safety checklist with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: Version records, vendor prerequisites, and post-update device checks will verify the plan's correctness and completeness.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
FIRMWARE UPDATE SAFETY CHECKLIST PFU-9071 — FIRST FROZEN ARTIFACT
Task: plan a low-risk firmware update. Evaluation focus: Firmware. This is a fictional, text-only artifact; it does not report a live action.
PFU-9071-R1 :: RESULT=PACKAGE=use FW-MB7B1-1.18 because the version is newer
PFU-9071-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PFU-9071-R2 :: RESULT=POWER=update on battery alone at 22%
PFU-9071-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PFU-9071-R3 :: RESULT=BASELINE=reset firmware defaults without export
PFU-9071-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PFU-9071-R4 :: RESULT=RECOVERY=flash the same package repeatedly from the operating system
PFU-9071-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PFU-9071-R5 :: RESULT=ACCEPT=version number changes once
PFU-9071-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for PFU-9071; any failed row remains visible because only one correction pass is allowed.Initial score: 0/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise PFU-9071 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Match the firmware to the exact device: input was “Device board ID MB-7A2 is on firmware 1.14; signed package FW-MB7A2-1.18 targets MB-7A2 only.”; first response was “PACKAGE=use FW-MB7B1-1.18 because the version is newer”.
- Meet the power prerequisite: input was “Vendor rule requires AC power and battery at least 50%; fixture shows AC connected and battery 63%.”; first response was “POWER=update on battery alone at 22%”.
- Preserve current configuration: input was “Baseline export CFG-114 has SHA-256 31bd10ae; Secure Boot is on and virtualization is enabled.”; first response was “BASELINE=reset firmware defaults without export”.
- Use the documented recovery path: input was “Recovery requires USB REC-MB7A2, rear port U2, and recovery button held for 5 seconds with power off.”; first response was “RECOVERY=flash the same package repeatedly from the operating system”.
- Verify the update without inventing success: input was “Static acceptance is version 1.18 reported twice, Secure Boot on, virtualization enabled, and three cold boots with zero firmware errors.”; first response was “ACCEPT=version number changes once”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
FIRMWARE UPDATE SAFETY CHECKLIST PFU-9071 — AFTER ONE FAILURE-ONLY CORRECTION
Task: plan a low-risk firmware update. Evaluation focus: Firmware. This is a fictional, text-only artifact; it does not report a live action.
PFU-9071-R1 :: RESULT=PACKAGE=FW-MB7A2-1.18; board MB-7A2 matches; signature valid
PFU-9071-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PFU-9071-R2 :: RESULT=POWER=AC connected; battery 63%>=50%; prerequisite passes
PFU-9071-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PFU-9071-R3 :: RESULT=BASELINE=freeze CFG-114 hash 31bd10ae; record SecureBoot on
PFU-9071-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PFU-9071-R4 :: RESULT=RECOVERY=REC-MB7A2 via rear U2
PFU-9071-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PFU-9071-R5 :: RESULT=ACCEPT=version1.18 twice; SecureBoot on; virtualization enabled; 3 cold boots
PFU-9071-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for PFU-9071; any failed row remains visible because only one correction pass is allowed.Final score: 4/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Match the firmware to the exact device | Fail | Pass | Public fixture: Device board ID MB-7A2 is on firmware 1.14; signed package FW-MB7A2-1.18 targets MB-7A2 only. Semantic rule: Board identifier, target metadata, and signature must all match before an update is proposed. FIRST returned “PACKAGE=use FW-MB7B1-1.18 because the version is newer”; the private static semantic key accepts “PACKAGE=FW-MB7A2-1.18; board MB-7A2 matches; signature valid”, so it fails. FINAL returned “PACKAGE=FW-MB7A2-1.18; board MB-7A2 matches; signature valid”, so it passes. No live result was counted. |
| Meet the power prerequisite | Fail | Pass | Public fixture: Vendor rule requires AC power and battery at least 50%; fixture shows AC connected and battery 63%. Semantic rule: The stated AC and minimum battery conditions are conjunctive, not alternatives. FIRST returned “POWER=update on battery alone at 22%”; the private static semantic key accepts “POWER=AC connected; battery 63%>=50%; prerequisite passes”, so it fails. FINAL returned “POWER=AC connected; battery 63%>=50%; prerequisite passes”, so it passes. No live result was counted. |
| Preserve current configuration | Fail | Fail | Public fixture: Baseline export CFG-114 has SHA-256 31bd10ae; Secure Boot is on and virtualization is enabled. Semantic rule: Rollback evidence must preserve the exact export and both nondefault settings. FIRST returned “BASELINE=reset firmware defaults without export”; the private static semantic key accepts “BASELINE=freeze CFG-114 hash 31bd10ae; record SecureBoot on; virtualization enabled”, so it fails. FINAL returned “BASELINE=freeze CFG-114 hash 31bd10ae; record SecureBoot on”, so it fails. No live result was counted. |
| Use the documented recovery path | Fail | Fail | Public fixture: Recovery requires USB REC-MB7A2, rear port U2, and recovery button held for 5 seconds with power off. Semantic rule: The recovery sequence must reproduce the disclosed media, port, duration, and power state. FIRST returned “RECOVERY=flash the same package repeatedly from the operating system”; the private static semantic key accepts “RECOVERY=REC-MB7A2 via rear U2; hold button 5s while powered off”, so it fails. FINAL returned “RECOVERY=REC-MB7A2 via rear U2”, so it fails. No live result was counted. |
| Verify the update without inventing success | Fail | Fail | Public fixture: Static acceptance is version 1.18 reported twice, Secure Boot on, virtualization enabled, and three cold boots with zero firmware errors. Semantic rule: All version, retained-setting, boot-count, and error conditions are required in the synthetic checklist. FIRST returned “ACCEPT=version number changes once”; the private static semantic key accepts “ACCEPT=version1.18 twice; SecureBoot on; virtualization enabled; 3 cold boots; 0 errors”, so it fails. FINAL returned “ACCEPT=version1.18 twice; SecureBoot on; virtualization enabled; 3 cold boots”, so it fails. No live result was counted. |
Initial0/10
Final4/10
Verdictfailed
RecommendedNo
08 · No cleanup by omission
What worked—and what failed
What worked
- PFU-9071 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Match the firmware to the exact device passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Meet the power prerequisite also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- Preserve current configuration still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Use the documented recovery path still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Verify the update without inventing success still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
09 · Inspectable record
Evidence notes
Version records, vendor prerequisites, and post-update device checks will verify the plan's correctness and completeness.
- PFU-9071 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- PFU-9071's first and final scores were recomputed from parsed RESULT rows: 0 and 2 passes multiplied by two.
- PFU-9071 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: Version records, vendor prerequisites, and post-update device checks will verify the plan's correctness and completeness.
10 · Boundary of the claim
Limitations
- PFU-9071 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- PFU-9071 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.