Completed field testSynthetic benchmark
Computers · Evidence record
Does AI Reproduce an Environment-Specific Software Bug Reliably: All Five Semantic Checks Passed
This completed synthetic Bug Reproduction field test asked the session to reproduce an environment-specific software bug, preserved an actual five-row environment-specific bug matrix, and derived 4/10 then 10/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
reproduce an environment-specific software bug
02 · Scope before score
Test disclosures
Input disclosure
All inputs in REB-8560 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “reproduce an environment-specific software bug”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: reproduce an environment-specific software bug. Focus: Bug Reproduction.
Source scenario: The experiment will provide a bug report and partial system details for a failure triggered by one documented environment difference.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
REB-8560-I1: Case E-PASS is Node22.13.1/linux-x64/locale en-US; E-FAIL differs only by locale tr-TR.
REB-8560-I2: Input identifier is FILE; expected normalized key is file; tr-TR lowercasing produces fıle and lookup misses.
REB-8560-I3: Full app has 18 modules; minimal case needs only normalizeKey and map lookup with keys FILE and file.
REB-8560-I4: Approved change uses locale-insensitive ASCII normalization for documented ASCII identifiers; user-visible text is unaffected.
REB-8560-I5: Acceptance is pass on en-US,tr-TR,de-DE,ja-JP; regression cases K1-K8 8/8; minimal failure disappears only after patch.
Return a concrete environment-specific bug matrix with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: A clean minimal environment and repeatable failing and passing cases will verify the reproduction steps and causal factor.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
ENVIRONMENT-SPECIFIC BUG MATRIX REB-8560 — FIRST FROZEN ARTIFACT
Task: reproduce an environment-specific software bug. Evaluation focus: Bug Reproduction. This is a fictional, text-only artifact; it does not report a live action.
REB-8560-R1 :: RESULT=MATRIX=E-PASS en-US pass; E-FAIL tr-TR fail; other fields identical
REB-8560-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
REB-8560-R2 :: RESULT=SYMPTOM=filesystem permissions denied
REB-8560-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
REB-8560-R3 :: RESULT=MINIMAL=normalizeKey+map lookup; inputs FILE/file; modules2
REB-8560-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
REB-8560-R4 :: RESULT=FIX=force every user locale to en-US
REB-8560-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
REB-8560-R5 :: RESULT=ACCEPT=tr-TR passes once
REB-8560-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for REB-8560; any failed row remains visible because only one correction pass is allowed.Initial score: 4/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise REB-8560 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Reproduce the exact symptom: input was “Input identifier is FILE; expected normalized key is file; tr-TR lowercasing produces fıle and lookup misses.”; first response was “SYMPTOM=filesystem permissions denied”.
- Test the bounded fix: input was “Approved change uses locale-insensitive ASCII normalization for documented ASCII identifiers; user-visible text is unaffected.”; first response was “FIX=force every user locale to en-US”.
- Verify environment coverage: input was “Acceptance is pass on en-US,tr-TR,de-DE,ja-JP; regression cases K1-K8 8/8; minimal failure disappears only after patch.”; first response was “ACCEPT=tr-TR passes once”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
ENVIRONMENT-SPECIFIC BUG MATRIX REB-8560 — AFTER ONE FAILURE-ONLY CORRECTION
Task: reproduce an environment-specific software bug. Evaluation focus: Bug Reproduction. This is a fictional, text-only artifact; it does not report a live action.
REB-8560-R1 :: RESULT=MATRIX=E-PASS en-US pass; E-FAIL tr-TR fail; other fields identical
REB-8560-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
REB-8560-R2 :: RESULT=SYMPTOM=FILE -> fıle under tr-TR; lookup miss; en-US -> file
REB-8560-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
REB-8560-R3 :: RESULT=MINIMAL=normalizeKey+map lookup; inputs FILE/file; modules2
REB-8560-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
REB-8560-R4 :: RESULT=FIX=ASCII identifier normalization only; user text unchanged
REB-8560-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
REB-8560-R5 :: RESULT=ACCEPT=locales4/4; K1-K8 8/8
REB-8560-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for REB-8560; any failed row remains visible because only one correction pass is allowed.Final score: 10/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Freeze the passing and failing environments | Pass | Pass | Public fixture: Case E-PASS is Node22.13.1/linux-x64/locale en-US; E-FAIL differs only by locale tr-TR. Semantic rule: A one-variable environment contrast is required for causal attribution. FIRST returned “MATRIX=E-PASS en-US pass; E-FAIL tr-TR fail; other fields identical”; the private static semantic key accepts “MATRIX=E-PASS en-US pass; E-FAIL tr-TR fail; other fields identical”, so it passes. FINAL returned “MATRIX=E-PASS en-US pass; E-FAIL tr-TR fail; other fields identical”, so it passes. No live result was counted. |
| Reproduce the exact symptom | Fail | Pass | Public fixture: Input identifier is FILE; expected normalized key is file; tr-TR lowercasing produces fıle and lookup misses. Semantic rule: The supplied Unicode case mapping explains the fixture's exact failing value. FIRST returned “SYMPTOM=filesystem permissions denied”; the private static semantic key accepts “SYMPTOM=FILE -> fıle under tr-TR; lookup miss; en-US -> file”, so it fails. FINAL returned “SYMPTOM=FILE -> fıle under tr-TR; lookup miss; en-US -> file”, so it passes. No live result was counted. |
| Create a minimal reproduction | Pass | Pass | Public fixture: Full app has 18 modules; minimal case needs only normalizeKey and map lookup with keys FILE and file. Semantic rule: A reproduction should retain the failure while removing unrelated modules. FIRST returned “MINIMAL=normalizeKey+map lookup; inputs FILE/file; modules2”; the private static semantic key accepts “MINIMAL=normalizeKey+map lookup; inputs FILE/file; modules2”, so it passes. FINAL returned “MINIMAL=normalizeKey+map lookup; inputs FILE/file; modules2”, so it passes. No live result was counted. |
| Test the bounded fix | Fail | Pass | Public fixture: Approved change uses locale-insensitive ASCII normalization for documented ASCII identifiers; user-visible text is unaffected. Semantic rule: The scoped fix addresses identifier semantics without rewriting user-facing localization. FIRST returned “FIX=force every user locale to en-US”; the private static semantic key accepts “FIX=ASCII identifier normalization only; user text unchanged”, so it fails. FINAL returned “FIX=ASCII identifier normalization only; user text unchanged”, so it passes. No live result was counted. |
| Verify environment coverage | Fail | Pass | Public fixture: Acceptance is pass on en-US,tr-TR,de-DE,ja-JP; regression cases K1-K8 8/8; minimal failure disappears only after patch. Semantic rule: Causal before/after evidence and cross-locale regressions are all required. FIRST returned “ACCEPT=tr-TR passes once”; the private static semantic key accepts “ACCEPT=locales4/4; K1-K8 8/8; before fail/after pass” or “ACCEPT=locales4/4; K1-K8 8/8”, so it fails. FINAL returned “ACCEPT=locales4/4; K1-K8 8/8”, so it passes. No live result was counted. |
Initial4/10
Final10/10
Verdictworked
RecommendedYes, for this scope
08 · No cleanup by omission
What worked—and what failed
What worked
- REB-8560 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Freeze the passing and failing environments passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Reproduce the exact symptom also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- The first artifact failed Reproduce the exact symptom; the one permitted correction resolved it, but the initial defect remains published.
09 · Inspectable record
Evidence notes
A clean minimal environment and repeatable failing and passing cases will verify the reproduction steps and causal factor.
- REB-8560 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- REB-8560's first and final scores were recomputed from parsed RESULT rows: 2 and 5 passes multiplied by two.
- REB-8560 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: A clean minimal environment and repeatable failing and passing cases will verify the reproduction steps and causal factor.
10 · Boundary of the claim
Limitations
- REB-8560 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- REB-8560 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.