Completed field testSynthetic benchmark
Computers · Evidence record
An AI Migration Plan for a Synthetic Password Vault: One Verified Gap Remained
This completed synthetic Credential Hygiene field test asked the session to plan a safe password-vault migration, preserved an actual five-row password-vault migration reconciliation, and derived 4/10 then 8/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
plan a safe password-vault migration
02 · Scope before score
Test disclosures
Input disclosure
All inputs in MPV-3188 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “plan a safe password-vault migration”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: plan a safe password-vault migration. Focus: Credential Hygiene.
Source scenario: The experiment will use synthetic vault entries to evaluate migration, validation, cleanup, and rollback instructions.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
MPV-3188-I1: Synthetic vault PV-12 contains 240 logins, 12 secure notes, 6 payment cards, 8 TOTP seeds, and 5 file attachments; source manifest hash is 15a29c70.
MPV-3188-I2: Login L044 and L177 share site and username. L177 has newer password version v3 and a note unique to L044; policy keeps one login with both latest credential and note.
MPV-3188-I3: Migration uses encrypted archive PVE-12 with Argon2id profile A3; transfer medium T12 is offline. Plain CSV, clipboard transfer, and cloud upload are forbidden.
MPV-3188-I4: Frozen controls are TOTP codes for seeds T01, T04, T08 at time 12:00, and attachment hashes A1=1c2a, A2=44d0, A3=9e11, A4=7b82, A5=0f66.
MPV-3188-I5: Destination expects 239 logins after the one merge, 12 notes, 6 cards, 8 TOTP, 5 attachments, sample login checks L001/L044-177/L240, plus source PV-12 unchanged.
Return a concrete password-vault migration reconciliation with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: A source manifest, destination audit, and duplicate and missing-entry checks will verify the migration plan.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
PASSWORD-VAULT MIGRATION RECONCILIATION MPV-3188 — FIRST FROZEN ARTIFACT
Task: plan a safe password-vault migration. Evaluation focus: Credential Hygiene. This is a fictional, text-only artifact; it does not report a live action.
MPV-3188-R1 :: RESULT=SOURCE=logins240; notes12; cards6; TOTP8; attachments5; manifest15a29c70
MPV-3188-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
MPV-3188-R2 :: RESULT=DUPLICATE=delete L044 including its unique note
MPV-3188-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
MPV-3188-R3 :: RESULT=TRANSFER=PVE-12 encrypted Argon2idA3 via offline T12; CSV0; clipboard0; cloud0
MPV-3188-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
MPV-3188-R4 :: RESULT=CONTENT=attachment filenames look correct and TOTP is untested
MPV-3188-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
MPV-3188-R5 :: RESULT=ACCEPT=destination application opens
MPV-3188-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for MPV-3188; any failed row remains visible because only one correction pass is allowed.Initial score: 4/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise MPV-3188 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Resolve the seeded duplicate safely: input was “Login L044 and L177 share site and username. L177 has newer password version v3 and a note unique to L044; policy keeps one login with both latest credential and note.”; first response was “DUPLICATE=delete L044 including its unique note”.
- Validate TOTP and attachment payloads: input was “Frozen controls are TOTP codes for seeds T01, T04, T08 at time 12:00, and attachment hashes A1=1c2a, A2=44d0, A3=9e11, A4=7b82, A5=0f66.”; first response was “CONTENT=attachment filenames look correct and TOTP is untested”.
- Define destination and recovery acceptance: input was “Destination expects 239 logins after the one merge, 12 notes, 6 cards, 8 TOTP, 5 attachments, sample login checks L001/L044-177/L240, plus source PV-12 unchanged.”; first response was “ACCEPT=destination application opens”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
PASSWORD-VAULT MIGRATION RECONCILIATION MPV-3188 — AFTER ONE FAILURE-ONLY CORRECTION
Task: plan a safe password-vault migration. Evaluation focus: Credential Hygiene. This is a fictional, text-only artifact; it does not report a live action.
MPV-3188-R1 :: RESULT=SOURCE=logins240; notes12; cards6; TOTP8; attachments5; manifest15a29c70
MPV-3188-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
MPV-3188-R2 :: RESULT=DUPLICATE=merge L044+L177; credential v3; preserve L044 note; output count239 logins
MPV-3188-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
MPV-3188-R3 :: RESULT=TRANSFER=PVE-12 encrypted Argon2idA3 via offline T12; CSV0; clipboard0; cloud0
MPV-3188-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
MPV-3188-R4 :: RESULT=CONTENT=TOTP T01+T04+T08 match at12:00; attachments5/5 hashes1c2a,44d0,9e11,7b82,0f66
MPV-3188-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
MPV-3188-R5 :: RESULT=ACCEPT=logins239; notes12; cards6; TOTP8; attachments5; samples3/3
MPV-3188-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for MPV-3188; any failed row remains visible because only one correction pass is allowed.Final score: 8/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Reconcile every source record class | Pass | Pass | Public fixture: Synthetic vault PV-12 contains 240 logins, 12 secure notes, 6 payment cards, 8 TOTP seeds, and 5 file attachments; source manifest hash is 15a29c70. Semantic rule: Migration scope must retain all five record classes and the frozen source identity. FIRST returned “SOURCE=logins240; notes12; cards6; TOTP8; attachments5; manifest15a29c70”; the private static semantic key accepts “SOURCE=logins240; notes12; cards6; TOTP8; attachments5; manifest15a29c70”, so it passes. FINAL returned “SOURCE=logins240; notes12; cards6; TOTP8; attachments5; manifest15a29c70”, so it passes. No live result was counted. |
| Resolve the seeded duplicate safely | Fail | Pass | Public fixture: Login L044 and L177 share site and username. L177 has newer password version v3 and a note unique to L044; policy keeps one login with both latest credential and note. Semantic rule: Deduplication must retain the newest credential and unique metadata while reconciling the login count. FIRST returned “DUPLICATE=delete L044 including its unique note”; the private static semantic key accepts “DUPLICATE=merge L044+L177; credential v3; preserve L044 note; output count239 logins”, so it fails. FINAL returned “DUPLICATE=merge L044+L177; credential v3; preserve L044 note; output count239 logins”, so it passes. No live result was counted. |
| Keep export material protected | Pass | Pass | Public fixture: Migration uses encrypted archive PVE-12 with Argon2id profile A3; transfer medium T12 is offline. Plain CSV, clipboard transfer, and cloud upload are forbidden. Semantic rule: The disclosed encrypted offline route is the only permitted migration path. FIRST returned “TRANSFER=PVE-12 encrypted Argon2idA3 via offline T12; CSV0; clipboard0; cloud0”; the private static semantic key accepts “TRANSFER=PVE-12 encrypted Argon2idA3 via offline T12; CSV0; clipboard0; cloud0”, so it passes. FINAL returned “TRANSFER=PVE-12 encrypted Argon2idA3 via offline T12; CSV0; clipboard0; cloud0”, so it passes. No live result was counted. |
| Validate TOTP and attachment payloads | Fail | Pass | Public fixture: Frozen controls are TOTP codes for seeds T01, T04, T08 at time 12:00, and attachment hashes A1=1c2a, A2=44d0, A3=9e11, A4=7b82, A5=0f66. Semantic rule: Encrypted import success does not establish seed behavior or binary attachment identity. FIRST returned “CONTENT=attachment filenames look correct and TOTP is untested”; the private static semantic key accepts “CONTENT=TOTP T01+T04+T08 match at12:00; attachments5/5 hashes1c2a,44d0,9e11,7b82,0f66”, so it fails. FINAL returned “CONTENT=TOTP T01+T04+T08 match at12:00; attachments5/5 hashes1c2a,44d0,9e11,7b82,0f66”, so it passes. No live result was counted. |
| Define destination and recovery acceptance | Fail | Fail | Public fixture: Destination expects 239 logins after the one merge, 12 notes, 6 cards, 8 TOTP, 5 attachments, sample login checks L001/L044-177/L240, plus source PV-12 unchanged. Semantic rule: Every reconciled class, duplicate outcome, frozen sample, and source-preservation condition must pass. FIRST returned “ACCEPT=destination application opens”; the private static semantic key accepts “ACCEPT=logins239; notes12; cards6; TOTP8; attachments5; samples3/3; PV-12 unchanged”, so it fails. FINAL returned “ACCEPT=logins239; notes12; cards6; TOTP8; attachments5; samples3/3”, so it fails. No live result was counted. |
Initial4/10
Final8/10
Verdictworked
RecommendedYes, for this scope
08 · No cleanup by omission
What worked—and what failed
What worked
- MPV-3188 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Reconcile every source record class passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Resolve the seeded duplicate safely also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- Define destination and recovery acceptance still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
09 · Inspectable record
Evidence notes
A source manifest, destination audit, and duplicate and missing-entry checks will verify the migration plan.
- MPV-3188 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- MPV-3188's first and final scores were recomputed from parsed RESULT rows: 2 and 4 passes multiplied by two.
- MPV-3188 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: A source manifest, destination audit, and duplicate and missing-entry checks will verify the migration plan.
10 · Boundary of the claim
Limitations
- MPV-3188 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- MPV-3188 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.