Completed field testSynthetic benchmark
Computers · Evidence record
Does an AI Backup Design Survive a Clean Restore: Three Semantic Checks Still Failed
This completed synthetic Backups field test asked the session to design a backup that can actually be restored, preserved an actual five-row restorable backup design, and derived 0/10 then 4/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
design a backup that can actually be restored
02 · Scope before score
Test disclosures
Input disclosure
All inputs in DRB-8471 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “design a backup that can actually be restored”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: design a backup that can actually be restored. Focus: Backups.
Source scenario: The experiment will ask AI to design an encrypted backup routine for a small mixed-file home directory.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
DRB-8471-I1: Home set HOME-17 has 312 files in Docs, Photos, and Finance totaling 8.4 GB; source manifest hash is aa7201f4.
DRB-8471-I2: Archive must use AES-256; recovery secret REC-17 is stored separately and test operator has a verified copy.
DRB-8471-I3: Policy requires 7 daily, 4 weekly, and 3 monthly versions; oldest monthly is protected from pruning.
DRB-8471-I4: Frozen restore sample is D-014, D-099, P-101, F-002, F-018 plus the empty folder Docs/Templates.
DRB-8471-I5: Pass is 312/312 entries, zero hash mismatches, empty folder present, REC-17 unlock succeeds, and restore time under 25 minutes.
Return a concrete restorable backup design with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: A clean restore destination and checksum manifest will verify completeness, integrity, and recovery instructions.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
RESTORABLE BACKUP DESIGN DRB-8471 — FIRST FROZEN ARTIFACT
Task: design a backup that can actually be restored. Evaluation focus: Backups. This is a fictional, text-only artifact; it does not report a live action.
DRB-8471-R1 :: RESULT=SCOPE=back up Docs only
DRB-8471-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
DRB-8471-R2 :: RESULT=ENCRYPT=store REC-17 inside the archive
DRB-8471-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
DRB-8471-R3 :: RESULT=RETENTION=keep only the newest copy
DRB-8471-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
DRB-8471-R4 :: RESULT=SAMPLE=restore one convenient photo
DRB-8471-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
DRB-8471-R5 :: RESULT=ACCEPT=backup command exits without error
DRB-8471-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for DRB-8471; any failed row remains visible because only one correction pass is allowed.Initial score: 0/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise DRB-8471 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Cover the complete source manifest: input was “Home set HOME-17 has 312 files in Docs, Photos, and Finance totaling 8.4 GB; source manifest hash is aa7201f4.”; first response was “SCOPE=back up Docs only”.
- Use the declared encryption boundary: input was “Archive must use AES-256; recovery secret REC-17 is stored separately and test operator has a verified copy.”; first response was “ENCRYPT=store REC-17 inside the archive”.
- Meet version and retention rules: input was “Policy requires 7 daily, 4 weekly, and 3 monthly versions; oldest monthly is protected from pruning.”; first response was “RETENTION=keep only the newest copy”.
- Design a clean restore sample: input was “Frozen restore sample is D-014, D-099, P-101, F-002, F-018 plus the empty folder Docs/Templates.”; first response was “SAMPLE=restore one convenient photo”.
- Define restore acceptance: input was “Pass is 312/312 entries, zero hash mismatches, empty folder present, REC-17 unlock succeeds, and restore time under 25 minutes.”; first response was “ACCEPT=backup command exits without error”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
RESTORABLE BACKUP DESIGN DRB-8471 — AFTER ONE FAILURE-ONLY CORRECTION
Task: design a backup that can actually be restored. Evaluation focus: Backups. This is a fictional, text-only artifact; it does not report a live action.
DRB-8471-R1 :: RESULT=SCOPE=312 files; Docs+Photos+Finance; 8.4GB; manifest aa7201f4
DRB-8471-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
DRB-8471-R2 :: RESULT=ENCRYPT=AES-256; keep REC-17 separate; verify operator copy
DRB-8471-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
DRB-8471-R3 :: RESULT=RETENTION=daily7; weekly4; monthly3
DRB-8471-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
DRB-8471-R4 :: RESULT=SAMPLE=restore D-014,D-099,P-101,F-002,F-018
DRB-8471-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
DRB-8471-R5 :: RESULT=ACCEPT=312/312; hash mismatches0; empty folder present; REC-17 unlock
DRB-8471-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for DRB-8471; any failed row remains visible because only one correction pass is allowed.Final score: 4/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Cover the complete source manifest | Fail | Pass | Public fixture: Home set HOME-17 has 312 files in Docs, Photos, and Finance totaling 8.4 GB; source manifest hash is aa7201f4. Semantic rule: A restorable design must enumerate all declared folders, count, size, and manifest. FIRST returned “SCOPE=back up Docs only”; the private static semantic key accepts “SCOPE=312 files; Docs+Photos+Finance; 8.4GB; manifest aa7201f4”, so it fails. FINAL returned “SCOPE=312 files; Docs+Photos+Finance; 8.4GB; manifest aa7201f4”, so it passes. No live result was counted. |
| Use the declared encryption boundary | Fail | Pass | Public fixture: Archive must use AES-256; recovery secret REC-17 is stored separately and test operator has a verified copy. Semantic rule: Encrypted backup and independently accessible recovery material are both mandatory. FIRST returned “ENCRYPT=store REC-17 inside the archive”; the private static semantic key accepts “ENCRYPT=AES-256; keep REC-17 separate; verify operator copy”, so it fails. FINAL returned “ENCRYPT=AES-256; keep REC-17 separate; verify operator copy”, so it passes. No live result was counted. |
| Meet version and retention rules | Fail | Fail | Public fixture: Policy requires 7 daily, 4 weekly, and 3 monthly versions; oldest monthly is protected from pruning. Semantic rule: All three retention tiers and the protected version must survive pruning. FIRST returned “RETENTION=keep only the newest copy”; the private static semantic key accepts “RETENTION=daily7; weekly4; monthly3; protect oldest monthly”, so it fails. FINAL returned “RETENTION=daily7; weekly4; monthly3”, so it fails. No live result was counted. |
| Design a clean restore sample | Fail | Fail | Public fixture: Frozen restore sample is D-014, D-099, P-101, F-002, F-018 plus the empty folder Docs/Templates. Semantic rule: The exact multi-folder sample and empty directory test exercise more than archive creation. FIRST returned “SAMPLE=restore one convenient photo”; the private static semantic key accepts “SAMPLE=restore D-014,D-099,P-101,F-002,F-018; empty folder Docs/Templates present”, so it fails. FINAL returned “SAMPLE=restore D-014,D-099,P-101,F-002,F-018”, so it fails. No live result was counted. |
| Define restore acceptance | Fail | Fail | Public fixture: Pass is 312/312 entries, zero hash mismatches, empty folder present, REC-17 unlock succeeds, and restore time under 25 minutes. Semantic rule: Only an independently checked restore proves the backup design is usable. FIRST returned “ACCEPT=backup command exits without error”; the private static semantic key accepts “ACCEPT=312/312; hash mismatches0; empty folder present; REC-17 unlock; restore<25min”, so it fails. FINAL returned “ACCEPT=312/312; hash mismatches0; empty folder present; REC-17 unlock”, so it fails. No live result was counted. |
Initial0/10
Final4/10
Verdictfailed
RecommendedNo
08 · No cleanup by omission
What worked—and what failed
What worked
- DRB-8471 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Cover the complete source manifest passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Use the declared encryption boundary also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- Meet version and retention rules still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Design a clean restore sample still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Define restore acceptance still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
09 · Inspectable record
Evidence notes
A clean restore destination and checksum manifest will verify completeness, integrity, and recovery instructions.
- DRB-8471 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- DRB-8471's first and final scores were recomputed from parsed RESULT rows: 0 and 2 passes multiplied by two.
- DRB-8471 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: A clean restore destination and checksum manifest will verify completeness, integrity, and recovery instructions.
10 · Boundary of the claim
Limitations
- DRB-8471 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- DRB-8471 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.