Completed field testSynthetic benchmark
Computers · Evidence record
Will AI Automate Backup Verification Across Broken Fixtures: One Verified Gap Remained
This completed synthetic Backup Testing field test asked the session to automate backup verification, preserved an actual five-row read-only backup verification matrix, and derived 4/10 then 8/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
automate backup verification
02 · Scope before score
Test disclosures
Input disclosure
All inputs in ABV-0997 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “automate backup verification”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: automate backup verification. Focus: Backup Testing.
Source scenario: The experiment will ask AI to create a read-only verification routine for complete, incomplete, and altered backup fixtures.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
ABV-0997-I1: Fixture B-OK contains 120 files and manifest hash set M-OK; every file is readable.
ABV-0997-I2: Fixture B-MISS omits DOC-071 while its manifest still expects 120 entries.
ABV-0997-I3: Fixture B-ALTER has PHOTO-018 hash 0a11 instead of expected 7be2; size is unchanged.
ABV-0997-I4: Fixture B-LOCK has DB-004 present with expected hash metadata but read permission denied.
ABV-0997-I5: Policy forbids chmod, restore, rename, timestamp changes, and manifest edits; output is report VERIFY-01 only.
Return a concrete read-only backup verification matrix with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: A seeded fixture matrix will verify detection of missing, changed, unreadable, and intact files.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
READ-ONLY BACKUP VERIFICATION MATRIX ABV-0997 — FIRST FROZEN ARTIFACT
Task: automate backup verification. Evaluation focus: Backup Testing. This is a fictional, text-only artifact; it does not report a live action.
ABV-0997-R1 :: RESULT=B-OK=FAIL because no files were rewritten
ABV-0997-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
ABV-0997-R2 :: RESULT=B-MISS=FAIL; missing DOC-071; observed119 expected120
ABV-0997-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
ABV-0997-R3 :: RESULT=B-ALTER=PASS because the size matches
ABV-0997-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
ABV-0997-R4 :: RESULT=B-LOCK=FAIL; DB-004 unreadable; do not claim hash verified
ABV-0997-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
ABV-0997-R5 :: RESULT=SAFETY=chmod DB-004 and repair B-MISS during verification
ABV-0997-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for ABV-0997; any failed row remains visible because only one correction pass is allowed.Initial score: 4/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise ABV-0997 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Detect the intact fixture: input was “Fixture B-OK contains 120 files and manifest hash set M-OK; every file is readable.”; first response was “B-OK=FAIL because no files were rewritten”.
- Detect the altered file: input was “Fixture B-ALTER has PHOTO-018 hash 0a11 instead of expected 7be2; size is unchanged.”; first response was “B-ALTER=PASS because the size matches”.
- Keep verification non-mutating: input was “Policy forbids chmod, restore, rename, timestamp changes, and manifest edits; output is report VERIFY-01 only.”; first response was “SAFETY=chmod DB-004 and repair B-MISS during verification”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
READ-ONLY BACKUP VERIFICATION MATRIX ABV-0997 — AFTER ONE FAILURE-ONLY CORRECTION
Task: automate backup verification. Evaluation focus: Backup Testing. This is a fictional, text-only artifact; it does not report a live action.
ABV-0997-R1 :: RESULT=B-OK=PASS; files120/120; hashes120/120; unreadable0
ABV-0997-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
ABV-0997-R2 :: RESULT=B-MISS=FAIL; missing DOC-071; observed119 expected120
ABV-0997-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
ABV-0997-R3 :: RESULT=B-ALTER=FAIL
ABV-0997-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
ABV-0997-R4 :: RESULT=B-LOCK=FAIL; DB-004 unreadable; do not claim hash verified
ABV-0997-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
ABV-0997-R5 :: RESULT=SAFETY=read-only scan; write VERIFY-01 only
ABV-0997-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for ABV-0997; any failed row remains visible because only one correction pass is allowed.Final score: 8/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Detect the intact fixture | Fail | Pass | Public fixture: Fixture B-OK contains 120 files and manifest hash set M-OK; every file is readable. Semantic rule: The verifier must recognize a complete, readable, hash-matching fixture. FIRST returned “B-OK=FAIL because no files were rewritten”; the private static semantic key accepts “B-OK=PASS; files120/120; hashes120/120; unreadable0”, so it fails. FINAL returned “B-OK=PASS; files120/120; hashes120/120; unreadable0”, so it passes. No live result was counted. |
| Detect the missing file | Pass | Pass | Public fixture: Fixture B-MISS omits DOC-071 while its manifest still expects 120 entries. Semantic rule: The manifest difference identifies one exact missing entry. FIRST returned “B-MISS=FAIL; missing DOC-071; observed119 expected120”; the private static semantic key accepts “B-MISS=FAIL; missing DOC-071; observed119 expected120”, so it passes. FINAL returned “B-MISS=FAIL; missing DOC-071; observed119 expected120”, so it passes. No live result was counted. |
| Detect the altered file | Fail | Fail | Public fixture: Fixture B-ALTER has PHOTO-018 hash 0a11 instead of expected 7be2; size is unchanged. Semantic rule: Content integrity depends on the fixed hash, not file size alone. FIRST returned “B-ALTER=PASS because the size matches”; the private static semantic key accepts “B-ALTER=FAIL; PHOTO-018 hash0a11!=7be2”, so it fails. FINAL returned “B-ALTER=FAIL”, so it fails. No live result was counted. |
| Detect the unreadable file | Pass | Pass | Public fixture: Fixture B-LOCK has DB-004 present with expected hash metadata but read permission denied. Semantic rule: A verifier cannot count a hash it could not read and must expose the access failure. FIRST returned “B-LOCK=FAIL; DB-004 unreadable; do not claim hash verified”; the private static semantic key accepts “B-LOCK=FAIL; DB-004 unreadable; do not claim hash verified”, so it passes. FINAL returned “B-LOCK=FAIL; DB-004 unreadable; do not claim hash verified”, so it passes. No live result was counted. |
| Keep verification non-mutating | Fail | Pass | Public fixture: Policy forbids chmod, restore, rename, timestamp changes, and manifest edits; output is report VERIFY-01 only. Semantic rule: Verification must report defects without changing the backup under test. FIRST returned “SAFETY=chmod DB-004 and repair B-MISS during verification”; the private static semantic key accepts “SAFETY=read-only scan; write VERIFY-01 only; fixture hashes and timestamps unchanged” or “SAFETY=read-only scan; write VERIFY-01 only”, so it fails. FINAL returned “SAFETY=read-only scan; write VERIFY-01 only”, so it passes. No live result was counted. |
Initial4/10
Final8/10
Verdictworked
RecommendedYes, for this scope
08 · No cleanup by omission
What worked—and what failed
What worked
- ABV-0997 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Detect the intact fixture passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Detect the missing file also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- Detect the altered file still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
09 · Inspectable record
Evidence notes
A seeded fixture matrix will verify detection of missing, changed, unreadable, and intact files.
- ABV-0997 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- ABV-0997's first and final scores were recomputed from parsed RESULT rows: 2 and 4 passes multiplied by two.
- ABV-0997 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: A seeded fixture matrix will verify detection of missing, changed, unreadable, and intact files.
10 · Boundary of the claim
Limitations
- ABV-0997 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- ABV-0997 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.