{
  "category": "computers",
  "slug": "computers-build-reproducible-dev-environment",
  "title": "Would AI Produce a Reproducible Development Environment: The Correction Reached 6/10",
  "task": "build a reproducible development environment",
  "excerpt": "This completed synthetic Dev Environments field test asked the session to build a reproducible development environment, preserved an actual five-row reproducible development environment manifest, and derived 2/10 then 6/10 from task-specific semantic checks after one failure-only correction.",
  "tool": "Codex multi-agent session",
  "model": "Exact underlying model identifier not disclosed by the Codex session",
  "publishedAt": "2026-02-16T08:00:00+08:00",
  "durationMinutes": 0,
  "testMode": "Synthetic benchmark",
  "inputDisclosure": "All inputs in BRDE-9389 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.",
  "runDisclosure": "A Codex multi-agent session generated one text-only first artifact for “build a reproducible development environment”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.",
  "prompt": "Complete a bounded synthetic field test for: build a reproducible development environment. Focus: Dev Environments.\nSource scenario: The experiment will specify a small project's runtime, services, and editor-independent requirements for a clean environment definition.\nUse only these five public fictional inputs; the scoring answers are intentionally withheld:\nBRDE-9389-I1: Project requires Node 22.13.1, package manager 10.9.2, Python 3.12.4, and database 16.3.\nBRDE-9389-I2: Local services are app TCP3000, database TCP5432, and mail fixture TCP1025; all bind to loopback only.\nBRDE-9389-I3: Fixture seed SEED-41 creates 20 users and 60 orders; expected database checksum after seed is 5a12c0ef.\nBRDE-9389-I4: Build and tests use shell commands only; editor settings E1 are convenience defaults and cannot be required.\nBRDE-9389-I5: Acceptance is setup A and B with identical tool versions, lock hash 44d18f00, data hash 5a12c0ef, build hash 92af7710, and tests 18/18.\nReturn a concrete reproducible development environment manifest with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: Two clean setups and identical build and test outputs will verify reproducibility from the written configuration.",
  "firstResult": "REPRODUCIBLE DEVELOPMENT ENVIRONMENT MANIFEST BRDE-9389 — FIRST FROZEN ARTIFACT\nTask: build a reproducible development environment. Evaluation focus: Dev Environments. This is a fictional, text-only artifact; it does not report a live action.\nBRDE-9389-R1 :: RESULT=TOOLCHAIN=use latest versions\nBRDE-9389-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nBRDE-9389-R2 :: RESULT=SERVICES=bind all ports to 0.0.0.0\nBRDE-9389-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nBRDE-9389-R3 :: RESULT=DATA=generate random users on each setup\nBRDE-9389-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nBRDE-9389-R4 :: RESULT=EDITOR=E1 optional; build/test independent of editor\nBRDE-9389-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nBRDE-9389-R5 :: RESULT=ACCEPT=one developer machine works\nBRDE-9389-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nArtifact boundary: exactly five scored rows were frozen for BRDE-9389; any failed row remains visible because only one correction pass is allowed.",
  "correctionPrompt": "Revise BRDE-9389 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:\n- Pin the complete toolchain: input was “Project requires Node 22.13.1, package manager 10.9.2, Python 3.12.4, and database 16.3.”; first response was “TOOLCHAIN=use latest versions”.\n- Declare services and ports: input was “Local services are app TCP3000, database TCP5432, and mail fixture TCP1025; all bind to loopback only.”; first response was “SERVICES=bind all ports to 0.0.0.0”.\n- Seed deterministic data: input was “Fixture seed SEED-41 creates 20 users and 60 orders; expected database checksum after seed is 5a12c0ef.”; first response was “DATA=generate random users on each setup”.\n- Compare two clean setups: input was “Acceptance is setup A and B with identical tool versions, lock hash 44d18f00, data hash 5a12c0ef, build hash 92af7710, and tests 18/18.”; first response was “ACCEPT=one developer machine works”.\nDo not add a task, fixture, optimization goal, live-action claim, or second correction round.",
  "finalResult": "REPRODUCIBLE DEVELOPMENT ENVIRONMENT MANIFEST BRDE-9389 — AFTER ONE FAILURE-ONLY CORRECTION\nTask: build a reproducible development environment. Evaluation focus: Dev Environments. This is a fictional, text-only artifact; it does not report a live action.\nBRDE-9389-R1 :: RESULT=TOOLCHAIN=Node22.13.1; pm10.9.2; Python3.12.4; DB16.3\nBRDE-9389-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nBRDE-9389-R2 :: RESULT=SERVICES=3000+5432+1025 on 127.0.0.1 only\nBRDE-9389-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nBRDE-9389-R3 :: RESULT=DATA=SEED-41; users20; orders60\nBRDE-9389-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nBRDE-9389-R4 :: RESULT=EDITOR=E1 optional; build/test independent of editor\nBRDE-9389-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nBRDE-9389-R5 :: RESULT=ACCEPT=A+B versions equal; lock44d18f00; data5a12c0ef; build92af7710\nBRDE-9389-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nArtifact boundary: exactly five scored rows were frozen for BRDE-9389; any failed row remains visible because only one correction pass is allowed.",
  "checks": [
    {
      "name": "Pin the complete toolchain",
      "firstPass": false,
      "finalPass": true,
      "evidence": "Public fixture: Project requires Node 22.13.1, package manager 10.9.2, Python 3.12.4, and database 16.3. Semantic rule: Every disclosed component needs an exact version to make clean setups comparable. FIRST returned “TOOLCHAIN=use latest versions”; the private static semantic key accepts “TOOLCHAIN=Node22.13.1; pm10.9.2; Python3.12.4; DB16.3”, so it fails. FINAL returned “TOOLCHAIN=Node22.13.1; pm10.9.2; Python3.12.4; DB16.3”, so it passes. No live result was counted."
    },
    {
      "name": "Declare services and ports",
      "firstPass": false,
      "finalPass": true,
      "evidence": "Public fixture: Local services are app TCP3000, database TCP5432, and mail fixture TCP1025; all bind to loopback only. Semantic rule: The environment must expose the three required local services without widening their network scope. FIRST returned “SERVICES=bind all ports to 0.0.0.0”; the private static semantic key accepts “SERVICES=3000+5432+1025 on 127.0.0.1 only”, so it fails. FINAL returned “SERVICES=3000+5432+1025 on 127.0.0.1 only”, so it passes. No live result was counted."
    },
    {
      "name": "Seed deterministic data",
      "firstPass": false,
      "finalPass": false,
      "evidence": "Public fixture: Fixture seed SEED-41 creates 20 users and 60 orders; expected database checksum after seed is 5a12c0ef. Semantic rule: A fixed seed, counts, and checksum make the two environments comparable. FIRST returned “DATA=generate random users on each setup”; the private static semantic key accepts “DATA=SEED-41; users20; orders60; hash5a12c0ef”, so it fails. FINAL returned “DATA=SEED-41; users20; orders60”, so it fails. No live result was counted."
    },
    {
      "name": "Keep editor configuration optional",
      "firstPass": true,
      "finalPass": true,
      "evidence": "Public fixture: Build and tests use shell commands only; editor settings E1 are convenience defaults and cannot be required. Semantic rule: The source requirement explicitly separates reproducibility from editor choice. FIRST returned “EDITOR=E1 optional; build/test independent of editor”; the private static semantic key accepts “EDITOR=E1 optional; build/test independent of editor”, so it passes. FINAL returned “EDITOR=E1 optional; build/test independent of editor”, so it passes. No live result was counted."
    },
    {
      "name": "Compare two clean setups",
      "firstPass": false,
      "finalPass": false,
      "evidence": "Public fixture: Acceptance is setup A and B with identical tool versions, lock hash 44d18f00, data hash 5a12c0ef, build hash 92af7710, and tests 18/18. Semantic rule: Two independent clean manifests must match across tools, dependencies, data, build, and tests. FIRST returned “ACCEPT=one developer machine works”; the private static semantic key accepts “ACCEPT=A+B versions equal; lock44d18f00; data5a12c0ef; build92af7710; tests18/18”, so it fails. FINAL returned “ACCEPT=A+B versions equal; lock44d18f00; data5a12c0ef; build92af7710”, so it fails. No live result was counted."
    }
  ],
  "initialScore": 2,
  "score": 6,
  "verdict": "mixed",
  "recommended": false,
  "whatWorked": [
    "BRDE-9389 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.",
    "Pin the complete toolchain passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.",
    "Declare services and ports also passed its task-specific rule with the final answer left visible."
  ],
  "whatFailed": [
    "Seed deterministic data still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.",
    "Compare two clean setups still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence."
  ],
  "evidencePlan": "Two clean setups and identical build and test outputs will verify reproducibility from the written configuration.",
  "evidenceNotes": [
    "BRDE-9389 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.",
    "BRDE-9389's first and final scores were recomputed from parsed RESULT rows: 1 and 3 passes multiplied by two.",
    "BRDE-9389 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: Two clean setups and identical build and test outputs will verify reproducibility from the written configuration."
  ],
  "limitations": [
    "BRDE-9389 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.",
    "BRDE-9389 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result."
  ]
}