Completed field testSynthetic benchmark
Computers · Evidence record
Ask AI to Find Duplicate Files—Without Touching the Originals: Three Semantic Checks Still Failed
This completed synthetic File Cleanup field test asked the session to find duplicate files without deleting originals, preserved an actual five-row duplicate file classification audit, and derived 0/10 then 4/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
find duplicate files without deleting originals
02 · Scope before score
Test disclosures
Input disclosure
All inputs in FDF-7561 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “find duplicate files without deleting originals”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: find duplicate files without deleting originals. Focus: File Cleanup.
Source scenario: The experiment will ask AI to classify exact and renamed duplicates in a synthetic personal-file collection.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
FDF-7561-I1: Source FDF-7561-SRC has 214 objects, 18 folders, 3.8 GB, and manifest hash 91bc02f4; it is read-only.
FDF-7561-I2: Objects FDF-7561-017 and FDF-7561-089 share content hash d4a1; FDF-7561-089 has newer metadata but no unique bytes.
FDF-7561-I3: Protected object FDF-7561-144 is 42 MB with hash 77ee10aa and must not be converted, moved, or deleted.
FDF-7561-I4: Expected destination is 213 unique byte streams, 214 metadata records, 18 folders, and zero hash mismatches.
FDF-7561-I5: Audit sample IDs are FDF-7561-001, FDF-7561-017, FDF-7561-089, FDF-7561-144, and FDF-7561-214; all five must open and match.
Return a concrete duplicate file classification audit with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: A known duplicate manifest and file hashes will verify precision while confirming that no source file changed.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
DUPLICATE FILE CLASSIFICATION AUDIT FDF-7561 — FIRST FROZEN ARTIFACT
Task: find duplicate files without deleting originals. Evaluation focus: File Cleanup. This is a fictional, text-only artifact; it does not report a live action.
FDF-7561-R1 :: RESULT=SOURCE=rename source objects during inventory
FDF-7561-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
FDF-7561-R2 :: RESULT=DUPLICATE=delete both objects
FDF-7561-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
FDF-7561-R3 :: RESULT=PROTECTED=convert FDF-7561-144 to save space
FDF-7561-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
FDF-7561-R4 :: RESULT=DESTINATION=214 byte streams and ignore metadata records
FDF-7561-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
FDF-7561-R5 :: RESULT=SAMPLE=check one convenient object
FDF-7561-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for FDF-7561; any failed row remains visible because only one correction pass is allowed.Initial score: 0/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise FDF-7561 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Inventory the source without mutation: input was “Source FDF-7561-SRC has 214 objects, 18 folders, 3.8 GB, and manifest hash 91bc02f4; it is read-only.”; first response was “SOURCE=rename source objects during inventory”.
- Handle the seeded duplicate or conflict: input was “Objects FDF-7561-017 and FDF-7561-089 share content hash d4a1; FDF-7561-089 has newer metadata but no unique bytes.”; first response was “DUPLICATE=delete both objects”.
- Preserve the protected item: input was “Protected object FDF-7561-144 is 42 MB with hash 77ee10aa and must not be converted, moved, or deleted.”; first response was “PROTECTED=convert FDF-7561-144 to save space”.
- Reconcile destination counts and hashes: input was “Expected destination is 213 unique byte streams, 214 metadata records, 18 folders, and zero hash mismatches.”; first response was “DESTINATION=214 byte streams and ignore metadata records”.
- Audit the frozen duplicate-file sample: input was “Audit sample IDs are FDF-7561-001, FDF-7561-017, FDF-7561-089, FDF-7561-144, and FDF-7561-214; all five must open and match.”; first response was “SAMPLE=check one convenient object”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
DUPLICATE FILE CLASSIFICATION AUDIT FDF-7561 — AFTER ONE FAILURE-ONLY CORRECTION
Task: find duplicate files without deleting originals. Evaluation focus: File Cleanup. This is a fictional, text-only artifact; it does not report a live action.
FDF-7561-R1 :: RESULT=SOURCE=read-only FDF-7561-SRC; 214 objects; 18 folders; 3.8 GB; hash 91bc02f4
FDF-7561-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
FDF-7561-R2 :: RESULT=DUPLICATE=retain FDF-7561-017 bytes; preserve FDF-7561-089 metadata in the exception ledger
FDF-7561-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
FDF-7561-R3 :: RESULT=PROTECTED=move FDF-7561-144 to review and mark hash UNVERIFIED
FDF-7561-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
FDF-7561-R4 :: RESULT=DESTINATION=213 byte streams; 18 folders; allow 1 hash mismatch
FDF-7561-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
FDF-7561-R5 :: RESULT=SAMPLE=check four IDs and skip FDF-7561-144
FDF-7561-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for FDF-7561; any failed row remains visible because only one correction pass is allowed.Final score: 4/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Inventory the source without mutation | Fail | Pass | Public fixture: Source FDF-7561-SRC has 214 objects, 18 folders, 3.8 GB, and manifest hash 91bc02f4; it is read-only. Semantic rule: The source must remain read-only and all four manifest facts must be preserved. FIRST returned “SOURCE=rename source objects during inventory”; the private static semantic key accepts “SOURCE=read-only FDF-7561-SRC; 214 objects; 18 folders; 3.8 GB; hash 91bc02f4”, so it fails. FINAL returned “SOURCE=read-only FDF-7561-SRC; 214 objects; 18 folders; 3.8 GB; hash 91bc02f4”, so it passes. No live result was counted. |
| Handle the seeded duplicate or conflict | Fail | Pass | Public fixture: Objects FDF-7561-017 and FDF-7561-089 share content hash d4a1; FDF-7561-089 has newer metadata but no unique bytes. Semantic rule: Equal content hashes allow byte deduplication, but distinct metadata must remain auditable. FIRST returned “DUPLICATE=delete both objects”; the private static semantic key accepts “DUPLICATE=retain FDF-7561-017 bytes; preserve FDF-7561-089 metadata in the exception ledger”, so it fails. FINAL returned “DUPLICATE=retain FDF-7561-017 bytes; preserve FDF-7561-089 metadata in the exception ledger”, so it passes. No live result was counted. |
| Preserve the protected item | Fail | Fail | Public fixture: Protected object FDF-7561-144 is 42 MB with hash 77ee10aa and must not be converted, moved, or deleted. Semantic rule: The protected object has an explicit no-change rule and fixed hash. FIRST returned “PROTECTED=convert FDF-7561-144 to save space”; the private static semantic key accepts “PROTECTED=leave FDF-7561-144 at 42 MB with hash 77ee10aa unchanged”, so it fails. FINAL returned “PROTECTED=move FDF-7561-144 to review and mark hash UNVERIFIED”, so it fails. No live result was counted. |
| Reconcile destination counts and hashes | Fail | Fail | Public fixture: Expected destination is 213 unique byte streams, 214 metadata records, 18 folders, and zero hash mismatches. Semantic rule: The destination must satisfy each exact count and zero-integrity-error threshold. FIRST returned “DESTINATION=214 byte streams and ignore metadata records”; the private static semantic key accepts “DESTINATION=213 byte streams; 214 metadata records; 18 folders; 0 hash mismatches”, so it fails. FINAL returned “DESTINATION=213 byte streams; 18 folders; allow 1 hash mismatch”, so it fails. No live result was counted. |
| Audit the frozen duplicate-file sample | Fail | Fail | Public fixture: Audit sample IDs are FDF-7561-001, FDF-7561-017, FDF-7561-089, FDF-7561-144, and FDF-7561-214; all five must open and match. Semantic rule: The frozen five-ID audit sample is the minimum duplicate-classification acceptance set. FIRST returned “SAMPLE=check one convenient object”; the private static semantic key accepts “SAMPLE=FDF-7561-001,FDF-7561-017,FDF-7561-089,FDF-7561-144,FDF-7561-214 all open and match”, so it fails. FINAL returned “SAMPLE=check four IDs and skip FDF-7561-144”, so it fails. No live result was counted. |
Initial0/10
Final4/10
Verdictfailed
RecommendedNo
08 · No cleanup by omission
What worked—and what failed
What worked
- FDF-7561 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Inventory the source without mutation passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Handle the seeded duplicate or conflict also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- Preserve the protected item still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Reconcile destination counts and hashes still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Audit the frozen duplicate-file sample still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
09 · Inspectable record
Evidence notes
A known duplicate manifest and file hashes will verify precision while confirming that no source file changed.
- FDF-7561 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- FDF-7561's first and final scores were recomputed from parsed RESULT rows: 0 and 2 passes multiplied by two.
- FDF-7561 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: A known duplicate manifest and file hashes will verify precision while confirming that no source file changed.
10 · Boundary of the claim
Limitations
- FDF-7561 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- FDF-7561 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.