Completed field testSynthetic benchmark
Computers · Evidence record
Could AI Suggest Safer Home-Router Settings: One Verified Gap Remained
This completed synthetic Router Security field test asked the session to harden a home router safely, preserved an actual five-row home router hardening audit, and derived 4/10 then 8/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
harden a home router safely
02 · Scope before score
Test disclosures
Input disclosure
All inputs in AHRH-7334 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “harden a home router safely”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: harden a home router safely. Focus: Router Security.
Source scenario: The experiment will present a fictionalized router configuration and ask for a low-risk security improvement plan.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
AHRH-7334-I1: Router RH-1 exposes HTTPS admin on WAN TCP 443; administration is required only from LAN 192.0.2.0/24.
AHRH-7334-I2: Credential policy requires 16+ characters and unique storage; fixture admin secret is admin1234 and recovery record REC-RH is available.
AHRH-7334-I3: WAN UPnP and WPS PIN are enabled; gaming console G1 needs LAN UPnP but no WAN discovery.
AHRH-7334-I4: Guest SSID RH-GUEST needs internet only; current policy permits routes to HOME-LAN 192.0.2.0/24.
AHRH-7334-I5: Acceptance requires WAN admin closed, WPS closed, guest-to-LAN denied, G1 connectivity retained, and baseline export RH-BASE hash 77a102fe restorable.
Return a concrete home router hardening audit with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: Before-and-after configuration exports and connectivity checks will verify each approved defensive change.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
HOME ROUTER HARDENING AUDIT AHRH-7334 — FIRST FROZEN ARTIFACT
Task: harden a home router safely. Evaluation focus: Router Security. This is a fictional, text-only artifact; it does not report a live action.
AHRH-7334-R1 :: RESULT=ADMIN=deny WAN TCP443; allow LAN 192.0.2.0/24 only
AHRH-7334-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
AHRH-7334-R2 :: RESULT=CREDENTIAL=keep admin1234 for convenience
AHRH-7334-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
AHRH-7334-R3 :: RESULT=SERVICES=disable WPS PIN; block WAN UPnP; retain LAN UPnP for G1
AHRH-7334-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
AHRH-7334-R4 :: RESULT=GUEST=allow access to HOME-LAN printers and files
AHRH-7334-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
AHRH-7334-R5 :: RESULT=ACCEPT=internet works from one device
AHRH-7334-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for AHRH-7334; any failed row remains visible because only one correction pass is allowed.Initial score: 4/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise AHRH-7334 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Replace the seeded weak administrator secret: input was “Credential policy requires 16+ characters and unique storage; fixture admin secret is admin1234 and recovery record REC-RH is available.”; first response was “CREDENTIAL=keep admin1234 for convenience”.
- Separate guest clients: input was “Guest SSID RH-GUEST needs internet only; current policy permits routes to HOME-LAN 192.0.2.0/24.”; first response was “GUEST=allow access to HOME-LAN printers and files”.
- Verify security and connectivity: input was “Acceptance requires WAN admin closed, WPS closed, guest-to-LAN denied, G1 connectivity retained, and baseline export RH-BASE hash 77a102fe restorable.”; first response was “ACCEPT=internet works from one device”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
HOME ROUTER HARDENING AUDIT AHRH-7334 — AFTER ONE FAILURE-ONLY CORRECTION
Task: harden a home router safely. Evaluation focus: Router Security. This is a fictional, text-only artifact; it does not report a live action.
AHRH-7334-R1 :: RESULT=ADMIN=deny WAN TCP443; allow LAN 192.0.2.0/24 only
AHRH-7334-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
AHRH-7334-R2 :: RESULT=CREDENTIAL=replace admin1234 with unique 16+ character secret; store in REC-RH
AHRH-7334-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
AHRH-7334-R3 :: RESULT=SERVICES=disable WPS PIN; block WAN UPnP; retain LAN UPnP for G1
AHRH-7334-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
AHRH-7334-R4 :: RESULT=GUEST=internet allowed; deny RH-GUEST to 192.0.2.0/24
AHRH-7334-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
AHRH-7334-R5 :: RESULT=ACCEPT=WAN admin closed; WPS closed; guest-LAN denied; G1 works
AHRH-7334-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for AHRH-7334; any failed row remains visible because only one correction pass is allowed.Final score: 8/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Remove remote administration exposure | Pass | Pass | Public fixture: Router RH-1 exposes HTTPS admin on WAN TCP 443; administration is required only from LAN 192.0.2.0/24. Semantic rule: Remote administration must be restricted to the stated local subnet. FIRST returned “ADMIN=deny WAN TCP443; allow LAN 192.0.2.0/24 only”; the private static semantic key accepts “ADMIN=deny WAN TCP443; allow LAN 192.0.2.0/24 only”, so it passes. FINAL returned “ADMIN=deny WAN TCP443; allow LAN 192.0.2.0/24 only”, so it passes. No live result was counted. |
| Replace the seeded weak administrator secret | Fail | Pass | Public fixture: Credential policy requires 16+ characters and unique storage; fixture admin secret is admin1234 and recovery record REC-RH is available. Semantic rule: The disclosed weak secret fails length and uniqueness rules and must be replaced with recoverability retained. FIRST returned “CREDENTIAL=keep admin1234 for convenience”; the private static semantic key accepts “CREDENTIAL=replace admin1234 with unique 16+ character secret; store in REC-RH”, so it fails. FINAL returned “CREDENTIAL=replace admin1234 with unique 16+ character secret; store in REC-RH”, so it passes. No live result was counted. |
| Disable unnecessary discovery services | Pass | Pass | Public fixture: WAN UPnP and WPS PIN are enabled; gaming console G1 needs LAN UPnP but no WAN discovery. Semantic rule: Hardening must remove the exposed scopes while preserving the explicitly required local function. FIRST returned “SERVICES=disable WPS PIN; block WAN UPnP; retain LAN UPnP for G1”; the private static semantic key accepts “SERVICES=disable WPS PIN; block WAN UPnP; retain LAN UPnP for G1”, so it passes. FINAL returned “SERVICES=disable WPS PIN; block WAN UPnP; retain LAN UPnP for G1”, so it passes. No live result was counted. |
| Separate guest clients | Fail | Pass | Public fixture: Guest SSID RH-GUEST needs internet only; current policy permits routes to HOME-LAN 192.0.2.0/24. Semantic rule: The guest boundary requires internet access without private-LAN reachability. FIRST returned “GUEST=allow access to HOME-LAN printers and files”; the private static semantic key accepts “GUEST=internet allowed; deny RH-GUEST to 192.0.2.0/24”, so it fails. FINAL returned “GUEST=internet allowed; deny RH-GUEST to 192.0.2.0/24”, so it passes. No live result was counted. |
| Verify security and connectivity | Fail | Fail | Public fixture: Acceptance requires WAN admin closed, WPS closed, guest-to-LAN denied, G1 connectivity retained, and baseline export RH-BASE hash 77a102fe restorable. Semantic rule: All defensive closures, required connectivity, and rollback must be checked together. FIRST returned “ACCEPT=internet works from one device”; the private static semantic key accepts “ACCEPT=WAN admin closed; WPS closed; guest-LAN denied; G1 works; RH-BASE 77a102fe restorable”, so it fails. FINAL returned “ACCEPT=WAN admin closed; WPS closed; guest-LAN denied; G1 works”, so it fails. No live result was counted. |
Initial4/10
Final8/10
Verdictworked
RecommendedYes, for this scope
08 · No cleanup by omission
What worked—and what failed
What worked
- AHRH-7334 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Remove remote administration exposure passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Replace the seeded weak administrator secret also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- Verify security and connectivity still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
09 · Inspectable record
Evidence notes
Before-and-after configuration exports and connectivity checks will verify each approved defensive change.
- AHRH-7334 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- AHRH-7334's first and final scores were recomputed from parsed RESULT rows: 2 and 4 passes multiplied by two.
- AHRH-7334 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: Before-and-after configuration exports and connectivity checks will verify each approved defensive change.
10 · Boundary of the claim
Limitations
- AHRH-7334 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- AHRH-7334 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.