Completed field testSynthetic benchmark
Computers · Evidence record
How Well Might AI Audit Keyboard-Only Navigation: All Five Semantic Checks Passed
This completed synthetic Keyboard Access field test asked the session to audit desktop keyboard navigation, preserved an actual five-row desktop keyboard navigation trace, and derived 6/10 then 10/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
audit desktop keyboard navigation
02 · Scope before score
Test disclosures
Input disclosure
All inputs in AKN-4108 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “audit desktop keyboard navigation”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: audit desktop keyboard navigation. Focus: Keyboard Access.
Source scenario: The experiment will ask AI to assess a sample desktop interface for complete operation without a pointing device.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
AKN-4108-I1: Interface controls are New N1, Search S1, Results R1, Save B1; intended workflow order is N1→S1→R1→B1.
AKN-4108-I2: Opening Results R1 dialog sends focus to page body; expected first dialog control is Close D1 and Tab must stay in D1-D3.
AKN-4108-I3: Buttons N1/B1 use Enter and Space; link Help H1 uses Enter; Escape closes dialog without saving.
AKN-4108-I4: B1 focus outline is 1 px at 1.8:1; policy requires at least 2 px and 3:1.
AKN-4108-I5: Acceptance is task New→Search→Select→Save completed in under 12 Tab presses, modal trap works, focus visible, and mouse events zero.
Return a concrete desktop keyboard navigation trace with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: A recorded task walkthrough and focus-order checklist will verify every reported accessibility issue.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
DESKTOP KEYBOARD NAVIGATION TRACE AKN-4108 — FIRST FROZEN ARTIFACT
Task: audit desktop keyboard navigation. Evaluation focus: Keyboard Access. This is a fictional, text-only artifact; it does not report a live action.
AKN-4108-R1 :: RESULT=TAB_ORDER=N1>S1>R1>B1
AKN-4108-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
AKN-4108-R2 :: RESULT=DIALOG=focus D1 on open; cycle D1-D3; no page escape
AKN-4108-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
AKN-4108-R3 :: RESULT=KEYS=Enter+Space N1/B1; Enter H1; Escape close no-save
AKN-4108-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
AKN-4108-R4 :: RESULT=FOCUS=B1 passes because an outline exists
AKN-4108-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
AKN-4108-R5 :: RESULT=ACCEPT=Save activates by mouse
AKN-4108-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for AKN-4108; any failed row remains visible because only one correction pass is allowed.Initial score: 6/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise AKN-4108 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Expose the missing focus indicator: input was “B1 focus outline is 1 px at 1.8:1; policy requires at least 2 px and 3:1.”; first response was “FOCUS=B1 passes because an outline exists”.
- Complete the keyboard-only task: input was “Acceptance is task New→Search→Select→Save completed in under 12 Tab presses, modal trap works, focus visible, and mouse events zero.”; first response was “ACCEPT=Save activates by mouse”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
DESKTOP KEYBOARD NAVIGATION TRACE AKN-4108 — AFTER ONE FAILURE-ONLY CORRECTION
Task: audit desktop keyboard navigation. Evaluation focus: Keyboard Access. This is a fictional, text-only artifact; it does not report a live action.
AKN-4108-R1 :: RESULT=TAB_ORDER=N1>S1>R1>B1
AKN-4108-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
AKN-4108-R2 :: RESULT=DIALOG=focus D1 on open; cycle D1-D3; no page escape
AKN-4108-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
AKN-4108-R3 :: RESULT=KEYS=Enter+Space N1/B1; Enter H1; Escape close no-save
AKN-4108-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
AKN-4108-R4 :: RESULT=FOCUS=B1 fails1px/1.8:1; target>=2px and>=3:1
AKN-4108-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
AKN-4108-R5 :: RESULT=ACCEPT=task complete; Tab<=12; modal pass; focus pass; mouse0
AKN-4108-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for AKN-4108; any failed row remains visible because only one correction pass is allowed.Final score: 10/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Follow the visual task order | Pass | Pass | Public fixture: Interface controls are New N1, Search S1, Results R1, Save B1; intended workflow order is N1→S1→R1→B1. Semantic rule: Keyboard order must follow the declared task sequence and include every operable control. FIRST returned “TAB_ORDER=N1>S1>R1>B1”; the private static semantic key accepts “TAB_ORDER=N1>S1>R1>B1”, so it passes. FINAL returned “TAB_ORDER=N1>S1>R1>B1”, so it passes. No live result was counted. |
| Detect the focus trap defect | Pass | Pass | Public fixture: Opening Results R1 dialog sends focus to page body; expected first dialog control is Close D1 and Tab must stay in D1-D3. Semantic rule: Modal keyboard operation requires initial focus and containment. FIRST returned “DIALOG=focus D1 on open; cycle D1-D3; no page escape”; the private static semantic key accepts “DIALOG=focus D1 on open; cycle D1-D3; no page escape”, so it passes. FINAL returned “DIALOG=focus D1 on open; cycle D1-D3; no page escape”, so it passes. No live result was counted. |
| Verify activation keys | Pass | Pass | Public fixture: Buttons N1/B1 use Enter and Space; link Help H1 uses Enter; Escape closes dialog without saving. Semantic rule: Activation behavior must match the declared control types and escape rule. FIRST returned “KEYS=Enter+Space N1/B1; Enter H1; Escape close no-save”; the private static semantic key accepts “KEYS=Enter+Space N1/B1; Enter H1; Escape close no-save”, so it passes. FINAL returned “KEYS=Enter+Space N1/B1; Enter H1; Escape close no-save”, so it passes. No live result was counted. |
| Expose the missing focus indicator | Fail | Pass | Public fixture: B1 focus outline is 1 px at 1.8:1; policy requires at least 2 px and 3:1. Semantic rule: Both supplied thickness and contrast fall below the stated minimums. FIRST returned “FOCUS=B1 passes because an outline exists”; the private static semantic key accepts “FOCUS=B1 fails1px/1.8:1; target>=2px and>=3:1”, so it fails. FINAL returned “FOCUS=B1 fails1px/1.8:1; target>=2px and>=3:1”, so it passes. No live result was counted. |
| Complete the keyboard-only task | Fail | Pass | Public fixture: Acceptance is task New→Search→Select→Save completed in under 12 Tab presses, modal trap works, focus visible, and mouse events zero. Semantic rule: End-to-end completion, navigation efficiency, modal behavior, focus, and input modality all matter. FIRST returned “ACCEPT=Save activates by mouse”; the private static semantic key accepts “ACCEPT=task complete; Tab<=12; modal pass; focus pass; mouse0”, so it fails. FINAL returned “ACCEPT=task complete; Tab<=12; modal pass; focus pass; mouse0”, so it passes. No live result was counted. |
Initial6/10
Final10/10
Verdictworked
RecommendedYes, for this scope
08 · No cleanup by omission
What worked—and what failed
What worked
- AKN-4108 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Follow the visual task order passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Detect the focus trap defect also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- The first artifact failed Expose the missing focus indicator; the one permitted correction resolved it, but the initial defect remains published.
09 · Inspectable record
Evidence notes
A recorded task walkthrough and focus-order checklist will verify every reported accessibility issue.
- AKN-4108 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- AKN-4108's first and final scores were recomputed from parsed RESULT rows: 3 and 5 passes multiplied by two.
- AKN-4108 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: A recorded task walkthrough and focus-order checklist will verify every reported accessibility issue.
10 · Boundary of the claim
Limitations
- AKN-4108 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- AKN-4108 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.