Completed field testSynthetic benchmark
Computers · Evidence record
Is an AI-Configured Everyday Account Truly Least Privilege: Three Semantic Checks Still Failed
This completed synthetic Account Security field test asked the session to configure a safer everyday user account, preserved an actual five-row least-privilege account access matrix, and derived 0/10 then 4/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
configure a safer everyday user account
02 · Scope before score
Test disclosures
Input disclosure
All inputs in CSUA-5966 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “configure a safer everyday user account”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: configure a safer everyday user account. Focus: Account Security.
Source scenario: The experiment will ask for a least-privilege account setup that still supports ordinary work on a test computer.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
CSUA-5966-I1: Fixture PC-SA8 has owner-admin Rowan, new everyday user Casey, and disabled guest. Casey needs browser, office suite, printer, and personal folder but no system administration.
CSUA-5966-I2: Positive matrix for Casey is browser launch, edit Casey/Documents, print to PRN-1, and open office suite; installed application hashes are fixed.
CSUA-5966-I3: Negative matrix requires Casey cannot install unsigned app U7, edit system hosts file, read Rowan/Private, or add an administrator.
CSUA-5966-I4: Signed updater APP-4 may request Rowan credentials through the operating-system elevation prompt; Casey must never learn or store those credentials.
CSUA-5966-I5: Recovery record RK-SA8 is sealed for Rowan; acceptance requires Casey sign-in, four positive tests, four denials, logout isolation, and Rowan recovery sign-in.
Return a concrete least-privilege account access matrix with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: Permission checks and representative daily tasks will verify reduced privileges without blocking expected use.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
LEAST-PRIVILEGE ACCOUNT ACCESS MATRIX CSUA-5966 — FIRST FROZEN ARTIFACT
Task: configure a safer everyday user account. Evaluation focus: Account Security. This is a fictional, text-only artifact; it does not report a live action.
CSUA-5966-R1 :: RESULT=ROLES=make Casey an administrator for convenience
CSUA-5966-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
CSUA-5966-R2 :: RESULT=ALLOW=browser only and block personal document edits
CSUA-5966-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
CSUA-5966-R3 :: RESULT=DENY=block hosts edits but allow Rowan/Private read
CSUA-5966-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
CSUA-5966-R4 :: RESULT=UPDATE=share Rowan password with Casey
CSUA-5966-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
CSUA-5966-R5 :: RESULT=ACCEPT=Casey reaches the desktop once
CSUA-5966-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for CSUA-5966; any failed row remains visible because only one correction pass is allowed.Initial score: 0/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise CSUA-5966 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Assign the everyday account the standard role: input was “Fixture PC-SA8 has owner-admin Rowan, new everyday user Casey, and disabled guest. Casey needs browser, office suite, printer, and personal folder but no system administration.”; first response was “ROLES=make Casey an administrator for convenience”.
- Allow the declared daily tasks: input was “Positive matrix for Casey is browser launch, edit Casey/Documents, print to PRN-1, and open office suite; installed application hashes are fixed.”; first response was “ALLOW=browser only and block personal document edits”.
- Deny protected administrative actions: input was “Negative matrix requires Casey cannot install unsigned app U7, edit system hosts file, read Rowan/Private, or add an administrator.”; first response was “DENY=block hosts edits but allow Rowan/Private read”.
- Provide bounded elevation for updates: input was “Signed updater APP-4 may request Rowan credentials through the operating-system elevation prompt; Casey must never learn or store those credentials.”; first response was “UPDATE=share Rowan password with Casey”.
- Verify recovery without a bypass: input was “Recovery record RK-SA8 is sealed for Rowan; acceptance requires Casey sign-in, four positive tests, four denials, logout isolation, and Rowan recovery sign-in.”; first response was “ACCEPT=Casey reaches the desktop once”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
LEAST-PRIVILEGE ACCOUNT ACCESS MATRIX CSUA-5966 — AFTER ONE FAILURE-ONLY CORRECTION
Task: configure a safer everyday user account. Evaluation focus: Account Security. This is a fictional, text-only artifact; it does not report a live action.
CSUA-5966-R1 :: RESULT=ROLES=Rowan owner-admin; Casey standard; guest disabled; no second admin
CSUA-5966-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
CSUA-5966-R2 :: RESULT=ALLOW=browser+office+Casey/Documents write+PRN-1 print
CSUA-5966-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
CSUA-5966-R3 :: RESULT=DENY=U7 install+hosts edit blocked
CSUA-5966-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
CSUA-5966-R4 :: RESULT=UPDATE=APP-4 signed prompt to Rowan; Casey credential storage0
CSUA-5966-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
CSUA-5966-R5 :: RESULT=ACCEPT=Casey sign-in; positive4/4; denials4/4; logout isolation pass
CSUA-5966-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for CSUA-5966; any failed row remains visible because only one correction pass is allowed.Final score: 4/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Assign the everyday account the standard role | Fail | Pass | Public fixture: Fixture PC-SA8 has owner-admin Rowan, new everyday user Casey, and disabled guest. Casey needs browser, office suite, printer, and personal folder but no system administration. Semantic rule: The account role must satisfy daily tasks without duplicating the existing administrative authority. FIRST returned “ROLES=make Casey an administrator for convenience”; the private static semantic key accepts “ROLES=Rowan owner-admin; Casey standard; guest disabled; no second admin”, so it fails. FINAL returned “ROLES=Rowan owner-admin; Casey standard; guest disabled; no second admin”, so it passes. No live result was counted. |
| Allow the declared daily tasks | Fail | Fail | Public fixture: Positive matrix for Casey is browser launch, edit Casey/Documents, print to PRN-1, and open office suite; installed application hashes are fixed. Semantic rule: Least privilege remains usable only if every declared everyday action passes without changing applications. FIRST returned “ALLOW=browser only and block personal document edits”; the private static semantic key accepts “ALLOW=browser+office+Casey/Documents write+PRN-1 print; application hashes unchanged”, so it fails. FINAL returned “ALLOW=browser+office+Casey/Documents write+PRN-1 print”, so it fails. No live result was counted. |
| Deny protected administrative actions | Fail | Fail | Public fixture: Negative matrix requires Casey cannot install unsigned app U7, edit system hosts file, read Rowan/Private, or add an administrator. Semantic rule: Every named privileged or cross-user action is an explicit denial boundary. FIRST returned “DENY=block hosts edits but allow Rowan/Private read”; the private static semantic key accepts “DENY=U7 install+hosts edit blocked; Rowan/Private read+admin creation blocked”, so it fails. FINAL returned “DENY=U7 install+hosts edit blocked”, so it fails. No live result was counted. |
| Provide bounded elevation for updates | Fail | Pass | Public fixture: Signed updater APP-4 may request Rowan credentials through the operating-system elevation prompt; Casey must never learn or store those credentials. Semantic rule: Necessary updates use the existing administrator's interactive approval without weakening credential separation. FIRST returned “UPDATE=share Rowan password with Casey”; the private static semantic key accepts “UPDATE=APP-4 signed prompt to Rowan; Casey credential storage0; Casey remains standard” or “UPDATE=APP-4 signed prompt to Rowan; Casey credential storage0”, so it fails. FINAL returned “UPDATE=APP-4 signed prompt to Rowan; Casey credential storage0”, so it passes. No live result was counted. |
| Verify recovery without a bypass | Fail | Fail | Public fixture: Recovery record RK-SA8 is sealed for Rowan; acceptance requires Casey sign-in, four positive tests, four denials, logout isolation, and Rowan recovery sign-in. Semantic rule: Usability, privilege boundaries, session isolation, and authorized recovery must all be checked. FIRST returned “ACCEPT=Casey reaches the desktop once”; the private static semantic key accepts “ACCEPT=Casey sign-in; positive4/4; denials4/4; logout isolation pass; Rowan recovery pass”, so it fails. FINAL returned “ACCEPT=Casey sign-in; positive4/4; denials4/4; logout isolation pass”, so it fails. No live result was counted. |
Initial0/10
Final4/10
Verdictfailed
RecommendedNo
08 · No cleanup by omission
What worked—and what failed
What worked
- CSUA-5966 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Assign the everyday account the standard role passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Provide bounded elevation for updates also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- Allow the declared daily tasks still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Deny protected administrative actions still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Verify recovery without a bypass still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
09 · Inspectable record
Evidence notes
Permission checks and representative daily tasks will verify reduced privileges without blocking expected use.
- CSUA-5966 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- CSUA-5966's first and final scores were recomputed from parsed RESULT rows: 0 and 2 passes multiplied by two.
- CSUA-5966 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: Permission checks and representative daily tasks will verify reduced privileges without blocking expected use.
10 · Boundary of the claim
Limitations
- CSUA-5966 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- CSUA-5966 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.