Completed field testSynthetic benchmark
Computers · Evidence record
From One OS to Another: Porting Shell Automation: The Correction Reached 6/10
This completed synthetic Portability field test asked the session to port a shell automation between operating systems, preserved an actual five-row cross-platform shell portability matrix, and derived 0/10 then 6/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
port a shell automation between operating systems
02 · Scope before score
Test disclosures
Input disclosure
All inputs in PSA-0632 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “port a shell automation between operating systems”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: port a shell automation between operating systems. Focus: Portability.
Source scenario: The experiment will ask AI to adapt a file-processing automation for a second operating system under equivalent requirements.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
PSA-0632-I1: Script PORT-6 calls readlink -f on Linux; the target macOS runner lacks that option. Python 3.12 is installed on both systems and symlink fixture L1 must resolve to /tmp/Port Case/input.csv.
PSA-0632-I2: Linux GNU sed accepts sed -i, while macOS BSD sed requires an explicit backup suffix. Fixture C2 changes mode=dev to mode=test and must leave no backup file.
PSA-0632-I3: Input directory is /tmp/Port Case with files alpha one.csv and beta.csv; the current unquoted loop splits alpha one.csv into two arguments.
PSA-0632-I4: Worker W3 exits 23 after creating /tmp/port-6.stage; caller currently returns 0. Policy requires original nonzero status and removal of the stage file.
PSA-0632-I5: Frozen fixtures expect hashes a11c for F1, b22d for F2, and exit 23 for F3 on Ubuntu 24.04 and macOS 15; shellcheck findings must be zero.
Return a concrete cross-platform shell portability matrix with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: Identical fixtures and expected-output comparisons on both systems will verify compatible behavior and error handling.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
CROSS-PLATFORM SHELL PORTABILITY MATRIX PSA-0632 — FIRST FROZEN ARTIFACT
Task: port a shell automation between operating systems. Evaluation focus: Portability. This is a fictional, text-only artifact; it does not report a live action.
PSA-0632-R1 :: RESULT=PATH=retain readlink -f and assume macOS supports it
PSA-0632-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PSA-0632-R2 :: RESULT=EDIT=run sed -i identically on both systems
PSA-0632-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PSA-0632-R3 :: RESULT=QUOTING=escape only the directory name and leave loop variables unquoted
PSA-0632-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PSA-0632-R4 :: RESULT=FAILURE=return0 after printing a warning
PSA-0632-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PSA-0632-R5 :: RESULT=ACCEPT=one happy-path run on Ubuntu
PSA-0632-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for PSA-0632; any failed row remains visible because only one correction pass is allowed.Initial score: 0/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise PSA-0632 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Replace the nonportable path resolver: input was “Script PORT-6 calls readlink -f on Linux; the target macOS runner lacks that option. Python 3.12 is installed on both systems and symlink fixture L1 must resolve to /tmp/Port Case/input.csv.”; first response was “PATH=retain readlink -f and assume macOS supports it”.
- Handle in-place editing on both sed variants: input was “Linux GNU sed accepts sed -i, while macOS BSD sed requires an explicit backup suffix. Fixture C2 changes mode=dev to mode=test and must leave no backup file.”; first response was “EDIT=run sed -i identically on both systems”.
- Preserve paths containing spaces: input was “Input directory is /tmp/Port Case with files alpha one.csv and beta.csv; the current unquoted loop splits alpha one.csv into two arguments.”; first response was “QUOTING=escape only the directory name and leave loop variables unquoted”.
- Propagate failure and clean temporary state: input was “Worker W3 exits 23 after creating /tmp/port-6.stage; caller currently returns 0. Policy requires original nonzero status and removal of the stage file.”; first response was “FAILURE=return0 after printing a warning”.
- Define a two-system acceptance matrix: input was “Frozen fixtures expect hashes a11c for F1, b22d for F2, and exit 23 for F3 on Ubuntu 24.04 and macOS 15; shellcheck findings must be zero.”; first response was “ACCEPT=one happy-path run on Ubuntu”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
CROSS-PLATFORM SHELL PORTABILITY MATRIX PSA-0632 — AFTER ONE FAILURE-ONLY CORRECTION
Task: port a shell automation between operating systems. Evaluation focus: Portability. This is a fictional, text-only artifact; it does not report a live action.
PSA-0632-R1 :: RESULT=PATH=use python3.12 realpath equivalent; L1 resolves /tmp/Port Case/input.csv; Linux+macOS same
PSA-0632-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PSA-0632-R2 :: RESULT=EDIT=use portable temp-file then atomic replace; C2 mode=test; backup files0
PSA-0632-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PSA-0632-R3 :: RESULT=QUOTING=iterate two files; alpha one.csv remains one argument
PSA-0632-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PSA-0632-R4 :: RESULT=FAILURE=return23; remove /tmp/port-6.stage
PSA-0632-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PSA-0632-R5 :: RESULT=ACCEPT=Ubuntu+macOS F1=a11c F2=b22d F3=exit23; shellcheck0
PSA-0632-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for PSA-0632; any failed row remains visible because only one correction pass is allowed.Final score: 6/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Replace the nonportable path resolver | Fail | Pass | Public fixture: Script PORT-6 calls readlink -f on Linux; the target macOS runner lacks that option. Python 3.12 is installed on both systems and symlink fixture L1 must resolve to /tmp/Port Case/input.csv. Semantic rule: The replacement must use a disclosed common runtime and preserve the exact symlink resolution on both targets. FIRST returned “PATH=retain readlink -f and assume macOS supports it”; the private static semantic key accepts “PATH=use python3.12 realpath equivalent; L1 resolves /tmp/Port Case/input.csv; Linux+macOS same”, so it fails. FINAL returned “PATH=use python3.12 realpath equivalent; L1 resolves /tmp/Port Case/input.csv; Linux+macOS same”, so it passes. No live result was counted. |
| Handle in-place editing on both sed variants | Fail | Pass | Public fixture: Linux GNU sed accepts sed -i, while macOS BSD sed requires an explicit backup suffix. Fixture C2 changes mode=dev to mode=test and must leave no backup file. Semantic rule: The plan must avoid variant-specific flags while producing the same content and cleanup state. FIRST returned “EDIT=run sed -i identically on both systems”; the private static semantic key accepts “EDIT=use portable temp-file then atomic replace; C2 mode=test; backup files0”, so it fails. FINAL returned “EDIT=use portable temp-file then atomic replace; C2 mode=test; backup files0”, so it passes. No live result was counted. |
| Preserve paths containing spaces | Fail | Pass | Public fixture: Input directory is /tmp/Port Case with files alpha one.csv and beta.csv; the current unquoted loop splits alpha one.csv into two arguments. Semantic rule: Both the directory and each expanded filename require boundaries that survive embedded spaces. FIRST returned “QUOTING=escape only the directory name and leave loop variables unquoted”; the private static semantic key accepts “QUOTING=iterate two files; alpha one.csv remains one argument; word-splitting errors0” or “QUOTING=iterate two files; alpha one.csv remains one argument”, so it fails. FINAL returned “QUOTING=iterate two files; alpha one.csv remains one argument”, so it passes. No live result was counted. |
| Propagate failure and clean temporary state | Fail | Fail | Public fixture: Worker W3 exits 23 after creating /tmp/port-6.stage; caller currently returns 0. Policy requires original nonzero status and removal of the stage file. Semantic rule: The port must preserve the worker status, execute cleanup, and stop subsequent work. FIRST returned “FAILURE=return0 after printing a warning”; the private static semantic key accepts “FAILURE=return23; remove /tmp/port-6.stage; do not continue later workers”, so it fails. FINAL returned “FAILURE=return23; remove /tmp/port-6.stage”, so it fails. No live result was counted. |
| Define a two-system acceptance matrix | Fail | Fail | Public fixture: Frozen fixtures expect hashes a11c for F1, b22d for F2, and exit 23 for F3 on Ubuntu 24.04 and macOS 15; shellcheck findings must be zero. Semantic rule: Acceptance requires both operating systems, all three fixtures, identical hashes, the error path, and the static check. FIRST returned “ACCEPT=one happy-path run on Ubuntu”; the private static semantic key accepts “ACCEPT=Ubuntu+macOS F1=a11c F2=b22d F3=exit23; shellcheck0; outputs identical”, so it fails. FINAL returned “ACCEPT=Ubuntu+macOS F1=a11c F2=b22d F3=exit23; shellcheck0”, so it fails. No live result was counted. |
Initial0/10
Final6/10
Verdictmixed
RecommendedNo
08 · No cleanup by omission
What worked—and what failed
What worked
- PSA-0632 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Replace the nonportable path resolver passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Handle in-place editing on both sed variants also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- Propagate failure and clean temporary state still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Define a two-system acceptance matrix still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
09 · Inspectable record
Evidence notes
Identical fixtures and expected-output comparisons on both systems will verify compatible behavior and error handling.
- PSA-0632 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- PSA-0632's first and final scores were recomputed from parsed RESULT rows: 0 and 3 passes multiplied by two.
- PSA-0632 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: Identical fixtures and expected-output comparisons on both systems will verify compatible behavior and error handling.
10 · Boundary of the claim
Limitations
- PSA-0632 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- PSA-0632 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.