{
  "category": "computers",
  "slug": "computers-refactor-legacy-script",
  "title": "Putting a Fragile Legacy Script on AI's Refactoring Bench: Only One Semantic Check Held",
  "task": "refactor a fragile legacy script",
  "excerpt": "This completed synthetic Refactoring field test asked the session to refactor a fragile legacy script, preserved an actual five-row legacy script refactor diff, and derived 0/10 then 2/10 from task-specific semantic checks after one failure-only correction.",
  "tool": "Codex multi-agent session",
  "model": "Exact underlying model identifier not disclosed by the Codex session",
  "publishedAt": "2026-06-13T16:00:00+08:00",
  "durationMinutes": 0,
  "testMode": "Synthetic benchmark",
  "inputDisclosure": "All inputs in RLS-4274 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.",
  "runDisclosure": "A Codex multi-agent session generated one text-only first artifact for “refactor a fragile legacy script”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.",
  "prompt": "Complete a bounded synthetic field test for: refactor a fragile legacy script. Focus: Refactoring.\nSource scenario: The experiment will provide a small documented script with duplicated logic, weak error handling, and fixed behavior requirements.\nUse only these five public fictional inputs; the scoring answers are intentionally withheld:\nRLS-4274-I1: Script clean.js accepts --input, --output, and --dry-run; golden cases R1-R8 define stdout, stderr, files, and exit codes.\nRLS-4274-I2: Functions parseCsvA and parseCsvB contain 31 identical lines; A uses a comma delimiter and B uses a tab delimiter.\nRLS-4274-I3: normalizeInput maps null to missing, empty string to empty, and numeric zero to 0; case R4 covers all three.\nRLS-4274-I4: Unreadable input should emit ERR_READ, exit 3, and create no output; existing script currently throws a stack trace.\nRLS-4274-I5: Acceptance is R1-R8 8/8, duplicate parser block count 1, public flags unchanged, and static complexity below 9.\nReturn a concrete legacy script refactor diff with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: Regression tests and a static review will verify preserved behavior, clearer structure, and improved failure handling.",
  "firstResult": "LEGACY SCRIPT REFACTOR DIFF RLS-4274 — FIRST FROZEN ARTIFACT\nTask: refactor a fragile legacy script. Evaluation focus: Refactoring. This is a fictional, text-only artifact; it does not report a live action.\nRLS-4274-R1 :: RESULT=BEHAVIOR=rename --dry-run to --preview\nRLS-4274-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nRLS-4274-R2 :: RESULT=REFACTOR=delete parseCsvB and make both comma-delimited\nRLS-4274-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nRLS-4274-R3 :: RESULT=NORMALIZE=treat every falsy value as missing\nRLS-4274-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nRLS-4274-R4 :: RESULT=ERROR=catch every error and exit0\nRLS-4274-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nRLS-4274-R5 :: RESULT=ACCEPT=script has fewer lines\nRLS-4274-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nArtifact boundary: exactly five scored rows were frozen for RLS-4274; any failed row remains visible because only one correction pass is allowed.",
  "correctionPrompt": "Revise RLS-4274 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:\n- Preserve command behavior: input was “Script clean.js accepts --input, --output, and --dry-run; golden cases R1-R8 define stdout, stderr, files, and exit codes.”; first response was “BEHAVIOR=rename --dry-run to --preview”.\n- Consolidate duplicated parsing: input was “Functions parseCsvA and parseCsvB contain 31 identical lines; A uses a comma delimiter and B uses a tab delimiter.”; first response was “REFACTOR=delete parseCsvB and make both comma-delimited”.\n- Retain zero and empty-string semantics: input was “normalizeInput maps null to missing, empty string to empty, and numeric zero to 0; case R4 covers all three.”; first response was “NORMALIZE=treat every falsy value as missing”.\n- Add bounded error handling: input was “Unreadable input should emit ERR_READ, exit 3, and create no output; existing script currently throws a stack trace.”; first response was “ERROR=catch every error and exit0”.\n- Verify structure and regressions: input was “Acceptance is R1-R8 8/8, duplicate parser block count 1, public flags unchanged, and static complexity below 9.”; first response was “ACCEPT=script has fewer lines”.\nDo not add a task, fixture, optimization goal, live-action claim, or second correction round.",
  "finalResult": "LEGACY SCRIPT REFACTOR DIFF RLS-4274 — AFTER ONE FAILURE-ONLY CORRECTION\nTask: refactor a fragile legacy script. Evaluation focus: Refactoring. This is a fictional, text-only artifact; it does not report a live action.\nRLS-4274-R1 :: RESULT=BEHAVIOR=retain flags3; golden R1-R8 exact\nRLS-4274-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nRLS-4274-R2 :: RESULT=REFACTOR=one parseCsv(delimiter)\nRLS-4274-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nRLS-4274-R3 :: RESULT=NORMALIZE=null missing; empty stays empty\nRLS-4274-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nRLS-4274-R4 :: RESULT=ERROR=ERR_READ; exit3; output absent\nRLS-4274-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nRLS-4274-R5 :: RESULT=ACCEPT=R1-R8 8/8; parser block1; flags unchanged\nRLS-4274-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.\nArtifact boundary: exactly five scored rows were frozen for RLS-4274; any failed row remains visible because only one correction pass is allowed.",
  "checks": [
    {
      "name": "Preserve command behavior",
      "firstPass": false,
      "finalPass": true,
      "evidence": "Public fixture: Script clean.js accepts --input, --output, and --dry-run; golden cases R1-R8 define stdout, stderr, files, and exit codes. Semantic rule: Refactoring may change structure but not the documented command interface or eight outputs. FIRST returned “BEHAVIOR=rename --dry-run to --preview”; the private static semantic key accepts “BEHAVIOR=retain flags3; golden R1-R8 exact”, so it fails. FINAL returned “BEHAVIOR=retain flags3; golden R1-R8 exact”, so it passes. No live result was counted."
    },
    {
      "name": "Consolidate duplicated parsing",
      "firstPass": false,
      "finalPass": false,
      "evidence": "Public fixture: Functions parseCsvA and parseCsvB contain 31 identical lines; A uses a comma delimiter and B uses a tab delimiter. Semantic rule: The shared implementation must preserve the two disclosed delimiter arguments. FIRST returned “REFACTOR=delete parseCsvB and make both comma-delimited”; the private static semantic key accepts “REFACTOR=one parseCsv(delimiter); callers A comma and B tab”, so it fails. FINAL returned “REFACTOR=one parseCsv(delimiter)”, so it fails. No live result was counted."
    },
    {
      "name": "Retain zero and empty-string semantics",
      "firstPass": false,
      "finalPass": false,
      "evidence": "Public fixture: normalizeInput maps null to missing, empty string to empty, and numeric zero to 0; case R4 covers all three. Semantic rule: The regression fixture distinguishes three values that a broad truthy guard would conflate. FIRST returned “NORMALIZE=treat every falsy value as missing”; the private static semantic key accepts “NORMALIZE=null missing; empty stays empty; zero stays0”, so it fails. FINAL returned “NORMALIZE=null missing; empty stays empty”, so it fails. No live result was counted."
    },
    {
      "name": "Add bounded error handling",
      "firstPass": false,
      "finalPass": false,
      "evidence": "Public fixture: Unreadable input should emit ERR_READ, exit 3, and create no output; existing script currently throws a stack trace. Semantic rule: The documented failure contract specifies message, status, side effect, and disclosure. FIRST returned “ERROR=catch every error and exit0”; the private static semantic key accepts “ERROR=ERR_READ; exit3; output absent; no stack trace”, so it fails. FINAL returned “ERROR=ERR_READ; exit3; output absent”, so it fails. No live result was counted."
    },
    {
      "name": "Verify structure and regressions",
      "firstPass": false,
      "finalPass": false,
      "evidence": "Public fixture: Acceptance is R1-R8 8/8, duplicate parser block count 1, public flags unchanged, and static complexity below 9. Semantic rule: Behavioral, duplication, interface, and complexity gates jointly score the refactor. FIRST returned “ACCEPT=script has fewer lines”; the private static semantic key accepts “ACCEPT=R1-R8 8/8; parser block1; flags unchanged; complexity<9”, so it fails. FINAL returned “ACCEPT=R1-R8 8/8; parser block1; flags unchanged”, so it fails. No live result was counted."
    }
  ],
  "initialScore": 0,
  "score": 2,
  "verdict": "failed",
  "recommended": false,
  "whatWorked": [
    "RLS-4274 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.",
    "Preserve command behavior passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier."
  ],
  "whatFailed": [
    "Consolidate duplicated parsing still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.",
    "Retain zero and empty-string semantics still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.",
    "Add bounded error handling still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.",
    "Verify structure and regressions still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence."
  ],
  "evidencePlan": "Regression tests and a static review will verify preserved behavior, clearer structure, and improved failure handling.",
  "evidenceNotes": [
    "RLS-4274 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.",
    "RLS-4274's first and final scores were recomputed from parsed RESULT rows: 0 and 1 passes multiplied by two.",
    "RLS-4274 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: Regression tests and a static review will verify preserved behavior, clearer structure, and improved failure handling."
  ],
  "limitations": [
    "RLS-4274 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.",
    "RLS-4274 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result."
  ]
}