Completed field testSynthetic benchmark
Computers · Evidence record
How Should AI Harden a Home Router Without Breaking Devices: One Verified Gap Remained
This completed synthetic Router Hardening field test asked the session to harden a home router while preserving required device connectivity, preserved an actual five-row router hardening and legacy-compatibility matrix, and derived 4/10 then 8/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
harden a home router while preserving required device connectivity
02 · Scope before score
Test disclosures
Input disclosure
All inputs in HHR-0372 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “harden a home router while preserving required device connectivity”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: harden a home router while preserving required device connectivity. Focus: Router Hardening.
Source scenario: The experiment will use a lab router, documented client inventory, guest network requirement, legacy device constraint, and recovery procedure.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
HHR-0372-I1: Legacy sensor L1 supports WPA2-AES only; trusted clients support WPA3; guest isolation VLAN 30 is available.
HHR-0372-I2: WAN management is enabled on TCP 8443; administration is needed only from admin host 192.0.2.10.
HHR-0372-I3: Required matrix: laptop→printer TCP9100, phone→speaker TCP8009, guest→internet; guest→LAN must fail.
HHR-0372-I4: Baseline export RTR-22 hash 0c14fe77 and factory-reset recovery note FR-22 are verified.
HHR-0372-I5: Acceptance is zero WAN admin ports, required matrix 3/3, guest isolation, L1 telemetry, and configuration restore in under 15 minutes.
Return a concrete router hardening and legacy-compatibility matrix with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: Configuration export, port scans, connectivity checks, isolation tests, and a factory-reset recovery drill will verify the plan.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
ROUTER HARDENING AND LEGACY-COMPATIBILITY MATRIX HHR-0372 — FIRST FROZEN ARTIFACT
Task: harden a home router while preserving required device connectivity. Evaluation focus: Router Hardening. This is a fictional, text-only artifact; it does not report a live action.
HHR-0372-R1 :: RESULT=WIFI=trusted WPA3; place L1 on isolated VLAN30 with WPA2-AES
HHR-0372-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
HHR-0372-R2 :: RESULT=ADMIN=leave WAN8443 open with a stronger password
HHR-0372-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
HHR-0372-R3 :: RESULT=MATRIX=laptop-printer9100 pass; phone-speaker8009 pass; guest-internet pass; guest-LAN deny
HHR-0372-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
HHR-0372-R4 :: RESULT=ROLLBACK=overwrite RTR-22 with the hardened config
HHR-0372-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
HHR-0372-R5 :: RESULT=ACCEPT=port scan is quieter
HHR-0372-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for HHR-0372; any failed row remains visible because only one correction pass is allowed.Initial score: 4/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise HHR-0372 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Remove the exposed administration path: input was “WAN management is enabled on TCP 8443; administration is needed only from admin host 192.0.2.10.”; first response was “ADMIN=leave WAN8443 open with a stronger password”.
- Use the frozen configuration rollback: input was “Baseline export RTR-22 hash 0c14fe77 and factory-reset recovery note FR-22 are verified.”; first response was “ROLLBACK=overwrite RTR-22 with the hardened config”.
- Define a complete hardening retest: input was “Acceptance is zero WAN admin ports, required matrix 3/3, guest isolation, L1 telemetry, and configuration restore in under 15 minutes.”; first response was “ACCEPT=port scan is quieter”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
ROUTER HARDENING AND LEGACY-COMPATIBILITY MATRIX HHR-0372 — AFTER ONE FAILURE-ONLY CORRECTION
Task: harden a home router while preserving required device connectivity. Evaluation focus: Router Hardening. This is a fictional, text-only artifact; it does not report a live action.
HHR-0372-R1 :: RESULT=WIFI=trusted WPA3; place L1 on isolated VLAN30 with WPA2-AES
HHR-0372-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
HHR-0372-R2 :: RESULT=ADMIN=deny WAN8443; allow 192.0.2.10 on LAN only
HHR-0372-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
HHR-0372-R3 :: RESULT=MATRIX=laptop-printer9100 pass; phone-speaker8009 pass; guest-internet pass; guest-LAN deny
HHR-0372-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
HHR-0372-R4 :: RESULT=ROLLBACK=retain RTR-22 hash0c14fe77 and FR-22 before changes
HHR-0372-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
HHR-0372-R5 :: RESULT=ACCEPT=WAN admin0; required3/3; guest-LAN denied; L1 reports
HHR-0372-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for HHR-0372; any failed row remains visible because only one correction pass is allowed.Final score: 8/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Retain the legacy device on a bounded segment | Pass | Pass | Public fixture: Legacy sensor L1 supports WPA2-AES only; trusted clients support WPA3; guest isolation VLAN 30 is available. Semantic rule: The design must preserve L1 without weakening the trusted network. FIRST returned “WIFI=trusted WPA3; place L1 on isolated VLAN30 with WPA2-AES”; the private static semantic key accepts “WIFI=trusted WPA3; place L1 on isolated VLAN30 with WPA2-AES”, so it passes. FINAL returned “WIFI=trusted WPA3; place L1 on isolated VLAN30 with WPA2-AES”, so it passes. No live result was counted. |
| Remove the exposed administration path | Fail | Pass | Public fixture: WAN management is enabled on TCP 8443; administration is needed only from admin host 192.0.2.10. Semantic rule: Least privilege requires restricting management by interface and source, not only credential strength. FIRST returned “ADMIN=leave WAN8443 open with a stronger password”; the private static semantic key accepts “ADMIN=deny WAN8443; allow 192.0.2.10 on LAN only”, so it fails. FINAL returned “ADMIN=deny WAN8443; allow 192.0.2.10 on LAN only”, so it passes. No live result was counted. |
| Preserve required client connectivity | Pass | Pass | Public fixture: Required matrix: laptop→printer TCP9100, phone→speaker TCP8009, guest→internet; guest→LAN must fail. Semantic rule: Every required positive path and the guest negative path must be represented. FIRST returned “MATRIX=laptop-printer9100 pass; phone-speaker8009 pass; guest-internet pass; guest-LAN deny”; the private static semantic key accepts “MATRIX=laptop-printer9100 pass; phone-speaker8009 pass; guest-internet pass; guest-LAN deny”, so it passes. FINAL returned “MATRIX=laptop-printer9100 pass; phone-speaker8009 pass; guest-internet pass; guest-LAN deny”, so it passes. No live result was counted. |
| Use the frozen configuration rollback | Fail | Pass | Public fixture: Baseline export RTR-22 hash 0c14fe77 and factory-reset recovery note FR-22 are verified. Semantic rule: Both configuration rollback and reset recovery are required safeguards. FIRST returned “ROLLBACK=overwrite RTR-22 with the hardened config”; the private static semantic key accepts “ROLLBACK=retain RTR-22 hash0c14fe77 and FR-22 before changes”, so it fails. FINAL returned “ROLLBACK=retain RTR-22 hash0c14fe77 and FR-22 before changes”, so it passes. No live result was counted. |
| Define a complete hardening retest | Fail | Fail | Public fixture: Acceptance is zero WAN admin ports, required matrix 3/3, guest isolation, L1 telemetry, and configuration restore in under 15 minutes. Semantic rule: Security, compatibility, isolation, legacy function, and recovery time are all fixed gates. FIRST returned “ACCEPT=port scan is quieter”; the private static semantic key accepts “ACCEPT=WAN admin0; required3/3; guest-LAN denied; L1 reports; restore<15min”, so it fails. FINAL returned “ACCEPT=WAN admin0; required3/3; guest-LAN denied; L1 reports”, so it fails. No live result was counted. |
Initial4/10
Final8/10
Verdictworked
RecommendedYes, for this scope
08 · No cleanup by omission
What worked—and what failed
What worked
- HHR-0372 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Retain the legacy device on a bounded segment passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Remove the exposed administration path also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- Define a complete hardening retest still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
09 · Inspectable record
Evidence notes
Configuration export, port scans, connectivity checks, isolation tests, and a factory-reset recovery drill will verify the plan.
- HHR-0372 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- HHR-0372's first and final scores were recomputed from parsed RESULT rows: 2 and 4 passes multiplied by two.
- HHR-0372 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: Configuration export, port scans, connectivity checks, isolation tests, and a factory-reset recovery drill will verify the plan.
10 · Boundary of the claim
Limitations
- HHR-0372 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- HHR-0372 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.