Completed field testSynthetic benchmark
Computers · Evidence record
The Merge-Conflict Resolution Brief for AI: One Verified Gap Remained
This completed synthetic Version Control field test asked the session to resolve a complex merge conflict correctly, preserved an actual five-row three-way merge resolution record, and derived 2/10 then 8/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
resolve a complex merge conflict correctly
02 · Scope before score
Test disclosures
Input disclosure
All inputs in RMC-0354 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “resolve a complex merge conflict correctly”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: resolve a complex merge conflict correctly. Focus: Version Control.
Source scenario: The experiment will provide diverged branches whose conflicting edits encode documented behavioral requirements.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
RMC-0354-I1: Base config has retries=2 and region=us-east. Branch ours changes retries=4; branch theirs changes region=eu-west without touching retries.
RMC-0354-I2: Base timeout is 20s, ours proposes 45s, theirs proposes 30s. Release brief MR-27 fixes the approved timeout at 30s.
RMC-0354-I3: Base calls validate then save. Ours inserts normalize between them; theirs inserts audit after save. Both new calls are required and audit must observe saved state.
RMC-0354-I4: Both branches conflict in app.lock; merged package.json requires core 2.8.0 and audit-kit 1.3.0. Generator G9 deterministically produces lock hash 71cd09e2.
RMC-0354-I5: Expected gate is tests M01-M18 all pass, config snapshot hash 4b1d020f, lock hash 71cd09e2, and zero conflict markers.
Return a concrete three-way merge resolution record with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: The full test suite and a semantic diff against the intended resolution will verify the merged result.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
THREE-WAY MERGE RESOLUTION RECORD RMC-0354 — FIRST FROZEN ARTIFACT
Task: resolve a complex merge conflict correctly. Evaluation focus: Version Control. This is a fictional, text-only artifact; it does not report a live action.
RMC-0354-R1 :: RESULT=CONFIG=take ours wholesale and leave region us-east
RMC-0354-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
RMC-0354-R2 :: RESULT=TIMEOUT=average the branches to 37.5s
RMC-0354-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
RMC-0354-R3 :: RESULT=HANDLER=validate>audit>normalize>save
RMC-0354-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
RMC-0354-R4 :: RESULT=LOCK=regenerate with G9; core2.8.0+audit-kit1.3.0; hash71cd09e2; no hand merge
RMC-0354-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
RMC-0354-R5 :: RESULT=ACCEPT=compiler succeeds with two skipped tests
RMC-0354-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for RMC-0354; any failed row remains visible because only one correction pass is allowed.Initial score: 2/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise RMC-0354 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Preserve independent configuration edits: input was “Base config has retries=2 and region=us-east. Branch ours changes retries=4; branch theirs changes region=eu-west without touching retries.”; first response was “CONFIG=take ours wholesale and leave region us-east”.
- Resolve the actual timeout conflict by policy: input was “Base timeout is 20s, ours proposes 45s, theirs proposes 30s. Release brief MR-27 fixes the approved timeout at 30s.”; first response was “TIMEOUT=average the branches to 37.5s”.
- Merge the handler sequence semantically: input was “Base calls validate then save. Ours inserts normalize between them; theirs inserts audit after save. Both new calls are required and audit must observe saved state.”; first response was “HANDLER=validate>audit>normalize>save”.
- Meet the semantic acceptance gate: input was “Expected gate is tests M01-M18 all pass, config snapshot hash 4b1d020f, lock hash 71cd09e2, and zero conflict markers.”; first response was “ACCEPT=compiler succeeds with two skipped tests”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
THREE-WAY MERGE RESOLUTION RECORD RMC-0354 — AFTER ONE FAILURE-ONLY CORRECTION
Task: resolve a complex merge conflict correctly. Evaluation focus: Version Control. This is a fictional, text-only artifact; it does not report a live action.
RMC-0354-R1 :: RESULT=CONFIG=retries4 from ours; region eu-west from theirs; retain both independent edits
RMC-0354-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
RMC-0354-R2 :: RESULT=TIMEOUT=30s; source=MR-27; reject ours45s
RMC-0354-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
RMC-0354-R3 :: RESULT=HANDLER=validate>normalize>save>audit
RMC-0354-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
RMC-0354-R4 :: RESULT=LOCK=regenerate with G9; core2.8.0+audit-kit1.3.0; hash71cd09e2; no hand merge
RMC-0354-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
RMC-0354-R5 :: RESULT=ACCEPT=M01-M18 18/18; config4b1d020f; lock71cd09e2
RMC-0354-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for RMC-0354; any failed row remains visible because only one correction pass is allowed.Final score: 8/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Preserve independent configuration edits | Fail | Pass | Public fixture: Base config has retries=2 and region=us-east. Branch ours changes retries=4; branch theirs changes region=eu-west without touching retries. Semantic rule: A semantic merge must preserve nonoverlapping changes from both descendants of the base. FIRST returned “CONFIG=take ours wholesale and leave region us-east”; the private static semantic key accepts “CONFIG=retries4 from ours; region eu-west from theirs; retain both independent edits”, so it fails. FINAL returned “CONFIG=retries4 from ours; region eu-west from theirs; retain both independent edits”, so it passes. No live result was counted. |
| Resolve the actual timeout conflict by policy | Fail | Pass | Public fixture: Base timeout is 20s, ours proposes 45s, theirs proposes 30s. Release brief MR-27 fixes the approved timeout at 30s. Semantic rule: The supplied release brief, not an invented compromise, determines the conflicting value. FIRST returned “TIMEOUT=average the branches to 37.5s”; the private static semantic key accepts “TIMEOUT=30s; source=MR-27; reject ours45s”, so it fails. FINAL returned “TIMEOUT=30s; source=MR-27; reject ours45s”, so it passes. No live result was counted. |
| Merge the handler sequence semantically | Fail | Pass | Public fixture: Base calls validate then save. Ours inserts normalize between them; theirs inserts audit after save. Both new calls are required and audit must observe saved state. Semantic rule: The order must preserve both branch intents and the stated post-save audit dependency. FIRST returned “HANDLER=validate>audit>normalize>save”; the private static semantic key accepts “HANDLER=validate>normalize>save>audit; include both additions” or “HANDLER=validate>normalize>save>audit”, so it fails. FINAL returned “HANDLER=validate>normalize>save>audit”, so it passes. No live result was counted. |
| Treat the generated lockfile as derived | Pass | Pass | Public fixture: Both branches conflict in app.lock; merged package.json requires core 2.8.0 and audit-kit 1.3.0. Generator G9 deterministically produces lock hash 71cd09e2. Semantic rule: The lockfile must be regenerated from the resolved manifest and match the frozen deterministic hash. FIRST returned “LOCK=regenerate with G9; core2.8.0+audit-kit1.3.0; hash71cd09e2; no hand merge”; the private static semantic key accepts “LOCK=regenerate with G9; core2.8.0+audit-kit1.3.0; hash71cd09e2; no hand merge”, so it passes. FINAL returned “LOCK=regenerate with G9; core2.8.0+audit-kit1.3.0; hash71cd09e2; no hand merge”, so it passes. No live result was counted. |
| Meet the semantic acceptance gate | Fail | Fail | Public fixture: Expected gate is tests M01-M18 all pass, config snapshot hash 4b1d020f, lock hash 71cd09e2, and zero conflict markers. Semantic rule: Compilation alone cannot replace the full tests, both hashes, and marker scan. FIRST returned “ACCEPT=compiler succeeds with two skipped tests”; the private static semantic key accepts “ACCEPT=M01-M18 18/18; config4b1d020f; lock71cd09e2; markers0”, so it fails. FINAL returned “ACCEPT=M01-M18 18/18; config4b1d020f; lock71cd09e2”, so it fails. No live result was counted. |
Initial2/10
Final8/10
Verdictworked
RecommendedYes, for this scope
08 · No cleanup by omission
What worked—and what failed
What worked
- RMC-0354 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Preserve independent configuration edits passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Resolve the actual timeout conflict by policy also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- Meet the semantic acceptance gate still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
09 · Inspectable record
Evidence notes
The full test suite and a semantic diff against the intended resolution will verify the merged result.
- RMC-0354 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- RMC-0354's first and final scores were recomputed from parsed RESULT rows: 1 and 4 passes multiplied by two.
- RMC-0354 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: The full test suite and a semantic diff against the intended resolution will verify the merged result.
10 · Boundary of the claim
Limitations
- RMC-0354 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- RMC-0354 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.