Completed field testSynthetic benchmark
Computers · Evidence record
Build a Cleanly Installable CLI Package: Three Semantic Checks Still Failed
This completed synthetic App Packaging field test asked the session to package a command-line application for clean installation, preserved an actual five-row cli package lifecycle record, and derived 0/10 then 4/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
package a command-line application for clean installation
02 · Scope before score
Test disclosures
Input disclosure
All inputs in PCLA-5811 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “package a command-line application for clean installation”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: package a command-line application for clean installation. Focus: App Packaging.
Source scenario: The experiment will ask AI to package a small application with versioned metadata, dependencies, and uninstall behavior.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
PCLA-5811-I1: Application tidy version 1.4.2 exposes command tidy through bin/tidy.js; metadata license is MIT and runtime is Node >=22.
PCLA-5811-I2: Package needs bin/tidy.js, lib/a.js, lib/b.js, README, and LICENSE; tests/, fixtures/, and .env are excluded.
PCLA-5811-I3: Runtime requires parse 3.2.1 and color 5.0.0; dev-only testlib 9.1.0 must not install for users.
PCLA-5811-I4: Upgrade fixture moves 1.4.1→1.4.2 while retaining config CFG-USER and replacing executable hash old11 with new42.
PCLA-5811-I5: Acceptance: clean install, tidy --version prints 1.4.2, sample output hash 6cf1, upgrade passes, uninstall removes command but retains CFG-USER.
Return a concrete cli package lifecycle record with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: Fresh install, upgrade, invocation, and uninstall checks will verify the package lifecycle on a clean system.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
CLI PACKAGE LIFECYCLE RECORD PCLA-5811 — FIRST FROZEN ARTIFACT
Task: package a command-line application for clean installation. Evaluation focus: App Packaging. This is a fictional, text-only artifact; it does not report a live action.
PCLA-5811-R1 :: RESULT=METADATA=omit version and entry point
PCLA-5811-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PCLA-5811-R2 :: RESULT=CONTENTS=publish the whole repository including .env
PCLA-5811-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PCLA-5811-R3 :: RESULT=DEPENDENCIES=ship testlib and floating versions
PCLA-5811-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PCLA-5811-R4 :: RESULT=UPGRADE=delete CFG-USER during upgrade
PCLA-5811-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PCLA-5811-R5 :: RESULT=ACCEPT=archive file exists
PCLA-5811-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for PCLA-5811; any failed row remains visible because only one correction pass is allowed.Initial score: 0/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise PCLA-5811 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Declare package identity and entry point: input was “Application tidy version 1.4.2 exposes command tidy through bin/tidy.js; metadata license is MIT and runtime is Node >=22.”; first response was “METADATA=omit version and entry point”.
- Include only runtime files: input was “Package needs bin/tidy.js, lib/a.js, lib/b.js, README, and LICENSE; tests/, fixtures/, and .env are excluded.”; first response was “CONTENTS=publish the whole repository including .env”.
- Pin production dependencies: input was “Runtime requires parse 3.2.1 and color 5.0.0; dev-only testlib 9.1.0 must not install for users.”; first response was “DEPENDENCIES=ship testlib and floating versions”.
- Preserve upgrade behavior: input was “Upgrade fixture moves 1.4.1→1.4.2 while retaining config CFG-USER and replacing executable hash old11 with new42.”; first response was “UPGRADE=delete CFG-USER during upgrade”.
- Verify install, invocation, and uninstall: input was “Acceptance: clean install, tidy --version prints 1.4.2, sample output hash 6cf1, upgrade passes, uninstall removes command but retains CFG-USER.”; first response was “ACCEPT=archive file exists”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
CLI PACKAGE LIFECYCLE RECORD PCLA-5811 — AFTER ONE FAILURE-ONLY CORRECTION
Task: package a command-line application for clean installation. Evaluation focus: App Packaging. This is a fictional, text-only artifact; it does not report a live action.
PCLA-5811-R1 :: RESULT=METADATA=tidy1.4.2; bin tidy->bin/tidy.js; MIT; Node>=22
PCLA-5811-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PCLA-5811-R2 :: RESULT=CONTENTS=include5 declared paths; exclude tests+fixtures+.env
PCLA-5811-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PCLA-5811-R3 :: RESULT=DEPENDENCIES=parse3.2.1+color5.0.0 runtime
PCLA-5811-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PCLA-5811-R4 :: RESULT=UPGRADE=1.4.1>1.4.2; retain CFG-USER
PCLA-5811-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
PCLA-5811-R5 :: RESULT=ACCEPT=install pass; version1.4.2; sample6cf1; upgrade pass; command removed
PCLA-5811-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for PCLA-5811; any failed row remains visible because only one correction pass is allowed.Final score: 4/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Declare package identity and entry point | Fail | Pass | Public fixture: Application tidy version 1.4.2 exposes command tidy through bin/tidy.js; metadata license is MIT and runtime is Node >=22. Semantic rule: A clean installer needs the exact identity, executable mapping, license, and runtime constraint. FIRST returned “METADATA=omit version and entry point”; the private static semantic key accepts “METADATA=tidy1.4.2; bin tidy->bin/tidy.js; MIT; Node>=22”, so it fails. FINAL returned “METADATA=tidy1.4.2; bin tidy->bin/tidy.js; MIT; Node>=22”, so it passes. No live result was counted. |
| Include only runtime files | Fail | Pass | Public fixture: Package needs bin/tidy.js, lib/a.js, lib/b.js, README, and LICENSE; tests/, fixtures/, and .env are excluded. Semantic rule: The frozen allowlist and sensitive-file denylist determine package contents. FIRST returned “CONTENTS=publish the whole repository including .env”; the private static semantic key accepts “CONTENTS=include5 declared paths; exclude tests+fixtures+.env”, so it fails. FINAL returned “CONTENTS=include5 declared paths; exclude tests+fixtures+.env”, so it passes. No live result was counted. |
| Pin production dependencies | Fail | Fail | Public fixture: Runtime requires parse 3.2.1 and color 5.0.0; dev-only testlib 9.1.0 must not install for users. Semantic rule: User installation should contain exact production dependencies only. FIRST returned “DEPENDENCIES=ship testlib and floating versions”; the private static semantic key accepts “DEPENDENCIES=parse3.2.1+color5.0.0 runtime; exclude testlib9.1.0”, so it fails. FINAL returned “DEPENDENCIES=parse3.2.1+color5.0.0 runtime”, so it fails. No live result was counted. |
| Preserve upgrade behavior | Fail | Fail | Public fixture: Upgrade fixture moves 1.4.1→1.4.2 while retaining config CFG-USER and replacing executable hash old11 with new42. Semantic rule: Version replacement must not destroy the declared user-owned configuration. FIRST returned “UPGRADE=delete CFG-USER during upgrade”; the private static semantic key accepts “UPGRADE=1.4.1>1.4.2; retain CFG-USER; executable hashnew42”, so it fails. FINAL returned “UPGRADE=1.4.1>1.4.2; retain CFG-USER”, so it fails. No live result was counted. |
| Verify install, invocation, and uninstall | Fail | Fail | Public fixture: Acceptance: clean install, tidy --version prints 1.4.2, sample output hash 6cf1, upgrade passes, uninstall removes command but retains CFG-USER. Semantic rule: The full package lifecycle includes installation, behavior, upgrade, and scoped removal. FIRST returned “ACCEPT=archive file exists”; the private static semantic key accepts “ACCEPT=install pass; version1.4.2; sample6cf1; upgrade pass; command removed; CFG-USER retained”, so it fails. FINAL returned “ACCEPT=install pass; version1.4.2; sample6cf1; upgrade pass; command removed”, so it fails. No live result was counted. |
Initial0/10
Final4/10
Verdictfailed
RecommendedNo
08 · No cleanup by omission
What worked—and what failed
What worked
- PCLA-5811 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Declare package identity and entry point passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Include only runtime files also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- Pin production dependencies still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Preserve upgrade behavior still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
- Verify install, invocation, and uninstall still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
09 · Inspectable record
Evidence notes
Fresh install, upgrade, invocation, and uninstall checks will verify the package lifecycle on a clean system.
- PCLA-5811 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- PCLA-5811's first and final scores were recomputed from parsed RESULT rows: 0 and 2 passes multiplied by two.
- PCLA-5811 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: Fresh install, upgrade, invocation, and uninstall checks will verify the package lifecycle on a clean system.
10 · Boundary of the claim
Limitations
- PCLA-5811 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- PCLA-5811 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.