Completed field testSynthetic benchmark
Computers · Evidence record
Putting a Fragile Legacy Script on AI's Refactoring Bench: Only One Semantic Check Held
This completed synthetic Refactoring field test asked the session to refactor a fragile legacy script, preserved an actual five-row legacy script refactor diff, and derived 0/10 then 2/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
refactor a fragile legacy script
02 · Scope before score
Test disclosures
Input disclosure
All inputs in RLS-4274 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “refactor a fragile legacy script”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: refactor a fragile legacy script. Focus: Refactoring.
Source scenario: The experiment will provide a small documented script with duplicated logic, weak error handling, and fixed behavior requirements.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
RLS-4274-I1: Script clean.js accepts --input, --output, and --dry-run; golden cases R1-R8 define stdout, stderr, files, and exit codes.
RLS-4274-I2: Functions parseCsvA and parseCsvB contain 31 identical lines; A uses a comma delimiter and B uses a tab delimiter.
RLS-4274-I3: normalizeInput maps null to missing, empty string to empty, and numeric zero to 0; case R4 covers all three.
RLS-4274-I4: Unreadable input should emit ERR_READ, exit 3, and create no output; existing script currently throws a stack trace.
RLS-4274-I5: Acceptance is R1-R8 8/8, duplicate parser block count 1, public flags unchanged, and static complexity below 9.
Return a concrete legacy script refactor diff with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: Regression tests and a static review will verify preserved behavior, clearer structure, and improved failure handling.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
LEGACY SCRIPT REFACTOR DIFF RLS-4274 — FIRST FROZEN ARTIFACT
Task: refactor a fragile legacy script. Evaluation focus: Refactoring. This is a fictional, text-only artifact; it does not report a live action.
RLS-4274-R1 :: RESULT=BEHAVIOR=rename --dry-run to --preview
RLS-4274-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
RLS-4274-R2 :: RESULT=REFACTOR=delete parseCsvB and make both comma-delimited
RLS-4274-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
RLS-4274-R3 :: RESULT=NORMALIZE=treat every falsy value as missing
RLS-4274-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
RLS-4274-R4 :: RESULT=ERROR=catch every error and exit0
RLS-4274-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
RLS-4274-R5 :: RESULT=ACCEPT=script has fewer lines
RLS-4274-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for RLS-4274; any failed row remains visible because only one correction pass is allowed.Initial score: 0/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise RLS-4274 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Preserve command behavior: input was “Script clean.js accepts --input, --output, and --dry-run; golden cases R1-R8 define stdout, stderr, files, and exit codes.”; first response was “BEHAVIOR=rename --dry-run to --preview”.
- Consolidate duplicated parsing: input was “Functions parseCsvA and parseCsvB contain 31 identical lines; A uses a comma delimiter and B uses a tab delimiter.”; first response was “REFACTOR=delete parseCsvB and make both comma-delimited”.
- Retain zero and empty-string semantics: input was “normalizeInput maps null to missing, empty string to empty, and numeric zero to 0; case R4 covers all three.”; first response was “NORMALIZE=treat every falsy value as missing”.
- Add bounded error handling: input was “Unreadable input should emit ERR_READ, exit 3, and create no output; existing script currently throws a stack trace.”; first response was “ERROR=catch every error and exit0”.
- Verify structure and regressions: input was “Acceptance is R1-R8 8/8, duplicate parser block count 1, public flags unchanged, and static complexity below 9.”; first response was “ACCEPT=script has fewer lines”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
LEGACY SCRIPT REFACTOR DIFF RLS-4274 — AFTER ONE FAILURE-ONLY CORRECTION
Task: refactor a fragile legacy script. Evaluation focus: Refactoring. This is a fictional, text-only artifact; it does not report a live action.
RLS-4274-R1 :: RESULT=BEHAVIOR=retain flags3; golden R1-R8 exact
RLS-4274-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
RLS-4274-R2 :: RESULT=REFACTOR=one parseCsv(delimiter)
RLS-4274-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
RLS-4274-R3 :: RESULT=NORMALIZE=null missing; empty stays empty
RLS-4274-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
RLS-4274-R4 :: RESULT=ERROR=ERR_READ; exit3; output absent
RLS-4274-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
RLS-4274-R5 :: RESULT=ACCEPT=R1-R8 8/8; parser block1; flags unchanged
RLS-4274-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for RLS-4274; any failed row remains visible because only one correction pass is allowed.Final score: 2/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Preserve command behavior | Fail | Pass | Public fixture: Script clean.js accepts --input, --output, and --dry-run; golden cases R1-R8 define stdout, stderr, files, and exit codes. Semantic rule: Refactoring may change structure but not the documented command interface or eight outputs. FIRST returned “BEHAVIOR=rename --dry-run to --preview”; the private static semantic key accepts “BEHAVIOR=retain flags3; golden R1-R8 exact”, so it fails. FINAL returned “BEHAVIOR=retain flags3; golden R1-R8 exact”, so it passes. No live result was counted. |
| Consolidate duplicated parsing | Fail | Fail | Public fixture: Functions parseCsvA and parseCsvB contain 31 identical lines; A uses a comma delimiter and B uses a tab delimiter. Semantic rule: The shared implementation must preserve the two disclosed delimiter arguments. FIRST returned “REFACTOR=delete parseCsvB and make both comma-delimited”; the private static semantic key accepts “REFACTOR=one parseCsv(delimiter); callers A comma and B tab”, so it fails. FINAL returned “REFACTOR=one parseCsv(delimiter)”, so it fails. No live result was counted. |
| Retain zero and empty-string semantics | Fail | Fail | Public fixture: normalizeInput maps null to missing, empty string to empty, and numeric zero to 0; case R4 covers all three. Semantic rule: The regression fixture distinguishes three values that a broad truthy guard would conflate. FIRST returned “NORMALIZE=treat every falsy value as missing”; the private static semantic key accepts “NORMALIZE=null missing; empty stays empty; zero stays0”, so it fails. FINAL returned “NORMALIZE=null missing; empty stays empty”, so it fails. No live result was counted. |
| Add bounded error handling | Fail | Fail | Public fixture: Unreadable input should emit ERR_READ, exit 3, and create no output; existing script currently throws a stack trace. Semantic rule: The documented failure contract specifies message, status, side effect, and disclosure. FIRST returned “ERROR=catch every error and exit0”; the private static semantic key accepts “ERROR=ERR_READ; exit3; output absent; no stack trace”, so it fails. FINAL returned “ERROR=ERR_READ; exit3; output absent”, so it fails. No live result was counted. |
| Verify structure and regressions | Fail | Fail | Public fixture: Acceptance is R1-R8 8/8, duplicate parser block count 1, public flags unchanged, and static complexity below 9. Semantic rule: Behavioral, duplication, interface, and complexity gates jointly score the refactor. FIRST returned “ACCEPT=script has fewer lines”; the private static semantic key accepts “ACCEPT=R1-R8 8/8; parser block1; flags unchanged; complexity<9”, so it fails. FINAL returned “ACCEPT=R1-R8 8/8; parser block1; flags unchanged”, so it fails. No live result was counted. |
Initial0/10
Final2/10
Verdictfailed
RecommendedNo
08 · No cleanup by omission
What worked—and what failed
What worked
- RLS-4274 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Preserve command behavior passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
What failed or remained weak
- Consolidate duplicated parsing still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Retain zero and empty-string semantics still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Add bounded error handling still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Verify structure and regressions still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
09 · Inspectable record
Evidence notes
Regression tests and a static review will verify preserved behavior, clearer structure, and improved failure handling.
- RLS-4274 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- RLS-4274's first and final scores were recomputed from parsed RESULT rows: 0 and 1 passes multiplied by two.
- RLS-4274 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: Regression tests and a static review will verify preserved behavior, clearer structure, and improved failure handling.
10 · Boundary of the claim
Limitations
- RLS-4274 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- RLS-4274 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.