Completed field testSynthetic benchmark
Computers · Evidence record
Do AI-Guided DNS Filters Respect Family-Safety Boundaries: Only One Semantic Check Held
This completed synthetic DNS Filtering field test asked the session to configure family-safe dns filtering, preserved an actual five-row family dns filter policy, and derived 0/10 then 2/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
configure family-safe dns filtering
02 · Scope before score
Test disclosures
Input disclosure
All inputs in CFD-7265 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “configure family-safe dns filtering”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: configure family-safe dns filtering. Focus: DNS Filtering.
Source scenario: The experiment will test a reversible network configuration for blocking a documented set of unsuitable domains while preserving normal access.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
CFD-7265-I1: Family LAN policy specifies resolvers 192.0.2.53 and 192.0.2.54; router currently advertises 192.0.2.1 only.
CFD-7265-I2: Allowed control school.example must resolve to 192.0.2.80; it is not on any block list.
CFD-7265-I3: Blocked fixtures bad-a.example and bad-b.example must return the local block response 192.0.2.99.
CFD-7265-I4: Client C7 has manual resolver 203.0.113.53; policy permits DNS only to the family resolver pair on TCP/UDP 53.
CFD-7265-I5: Baseline DNS export DNS-BASE hash 50ad221c is restorable; acceptance is 2 allowed checks, 4 blocked checks, and bypass denial.
Return a concrete family dns filter policy with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: Allowed-domain, blocked-domain, bypass, and rollback checks will verify the intended filtering boundaries.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
FAMILY DNS FILTER POLICY CFD-7265 — FIRST FROZEN ARTIFACT
Task: configure family-safe dns filtering. Evaluation focus: DNS Filtering. This is a fictional, text-only artifact; it does not report a live action.
CFD-7265-R1 :: RESULT=RESOLVERS=keep 192.0.2.1 only
CFD-7265-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
CFD-7265-R2 :: RESULT=ALLOW=block school.example with the unsuitable set
CFD-7265-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
CFD-7265-R3 :: RESULT=BLOCK=bad-a.example resolves publicly; bad-b.example untested
CFD-7265-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
CFD-7265-R4 :: RESULT=BYPASS=leave all external DNS reachable
CFD-7265-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
CFD-7265-R5 :: RESULT=ACCEPT=one blocked page screenshot
CFD-7265-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for CFD-7265; any failed row remains visible because only one correction pass is allowed.Initial score: 0/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise CFD-7265 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Apply the documented resolver pair: input was “Family LAN policy specifies resolvers 192.0.2.53 and 192.0.2.54; router currently advertises 192.0.2.1 only.”; first response was “RESOLVERS=keep 192.0.2.1 only”.
- Preserve the allowed-domain control: input was “Allowed control school.example must resolve to 192.0.2.80; it is not on any block list.”; first response was “ALLOW=block school.example with the unsuitable set”.
- Block the seeded unsuitable domains: input was “Blocked fixtures bad-a.example and bad-b.example must return the local block response 192.0.2.99.”; first response was “BLOCK=bad-a.example resolves publicly; bad-b.example untested”.
- Close the documented bypass: input was “Client C7 has manual resolver 203.0.113.53; policy permits DNS only to the family resolver pair on TCP/UDP 53.”; first response was “BYPASS=leave all external DNS reachable”.
- Retain rollback and checks: input was “Baseline DNS export DNS-BASE hash 50ad221c is restorable; acceptance is 2 allowed checks, 4 blocked checks, and bypass denial.”; first response was “ACCEPT=one blocked page screenshot”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
FAMILY DNS FILTER POLICY CFD-7265 — AFTER ONE FAILURE-ONLY CORRECTION
Task: configure family-safe dns filtering. Evaluation focus: DNS Filtering. This is a fictional, text-only artifact; it does not report a live action.
CFD-7265-R1 :: RESULT=RESOLVERS=advertise 192.0.2.53 and 192.0.2.54 to Family LAN
CFD-7265-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
CFD-7265-R2 :: RESULT=ALLOW=school.example
CFD-7265-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
CFD-7265-R3 :: RESULT=BLOCK=bad-a.example+bad-b.example
CFD-7265-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
CFD-7265-R4 :: RESULT=BYPASS=deny C7 DNS to 203.0.113.53
CFD-7265-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
CFD-7265-R5 :: RESULT=ACCEPT=DNS-BASE 50ad221c restorable; allowed2/2; blocked4/4
CFD-7265-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for CFD-7265; any failed row remains visible because only one correction pass is allowed.Final score: 2/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Apply the documented resolver pair | Fail | Pass | Public fixture: Family LAN policy specifies resolvers 192.0.2.53 and 192.0.2.54; router currently advertises 192.0.2.1 only. Semantic rule: Both declared filtering resolvers must be distributed to the intended LAN. FIRST returned “RESOLVERS=keep 192.0.2.1 only”; the private static semantic key accepts “RESOLVERS=advertise 192.0.2.53 and 192.0.2.54 to Family LAN”, so it fails. FINAL returned “RESOLVERS=advertise 192.0.2.53 and 192.0.2.54 to Family LAN”, so it passes. No live result was counted. |
| Preserve the allowed-domain control | Fail | Fail | Public fixture: Allowed control school.example must resolve to 192.0.2.80; it is not on any block list. Semantic rule: A clean allowed control must continue to resolve to its fixed address. FIRST returned “ALLOW=block school.example with the unsuitable set”; the private static semantic key accepts “ALLOW=school.example; address192.0.2.80”, so it fails. FINAL returned “ALLOW=school.example”, so it fails. No live result was counted. |
| Block the seeded unsuitable domains | Fail | Fail | Public fixture: Blocked fixtures bad-a.example and bad-b.example must return the local block response 192.0.2.99. Semantic rule: Both exact seeded domains must produce the declared block response. FIRST returned “BLOCK=bad-a.example resolves publicly; bad-b.example untested”; the private static semantic key accepts “BLOCK=bad-a.example+bad-b.example; response192.0.2.99”, so it fails. FINAL returned “BLOCK=bad-a.example+bad-b.example”, so it fails. No live result was counted. |
| Close the documented bypass | Fail | Fail | Public fixture: Client C7 has manual resolver 203.0.113.53; policy permits DNS only to the family resolver pair on TCP/UDP 53. Semantic rule: Filtering is ineffective unless the stated manual-resolver path is bounded. FIRST returned “BYPASS=leave all external DNS reachable”; the private static semantic key accepts “BYPASS=deny C7 DNS to 203.0.113.53; allow TCP/UDP53 only to family pair”, so it fails. FINAL returned “BYPASS=deny C7 DNS to 203.0.113.53”, so it fails. No live result was counted. |
| Retain rollback and checks | Fail | Fail | Public fixture: Baseline DNS export DNS-BASE hash 50ad221c is restorable; acceptance is 2 allowed checks, 4 blocked checks, and bypass denial. Semantic rule: The test must cover allowed, blocked, bypass, and rollback behavior. FIRST returned “ACCEPT=one blocked page screenshot”; the private static semantic key accepts “ACCEPT=DNS-BASE 50ad221c restorable; allowed2/2; blocked4/4; bypass denied”, so it fails. FINAL returned “ACCEPT=DNS-BASE 50ad221c restorable; allowed2/2; blocked4/4”, so it fails. No live result was counted. |
Initial0/10
Final2/10
Verdictfailed
RecommendedNo
08 · No cleanup by omission
What worked—and what failed
What worked
- CFD-7265 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Apply the documented resolver pair passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
What failed or remained weak
- Preserve the allowed-domain control still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Block the seeded unsuitable domains still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Close the documented bypass still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Retain rollback and checks still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
09 · Inspectable record
Evidence notes
Allowed-domain, blocked-domain, bypass, and rollback checks will verify the intended filtering boundaries.
- CFD-7265 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- CFD-7265's first and final scores were recomputed from parsed RESULT rows: 0 and 1 passes multiplied by two.
- CFD-7265 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: Allowed-domain, blocked-domain, bypass, and rollback checks will verify the intended filtering boundaries.
10 · Boundary of the claim
Limitations
- CFD-7265 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- CFD-7265 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.