Work · Evidence record
Could AI Build a Complete RFP Compliance Matrix — Three of Five Checks Passed
The completed WFT-038 synthetic field test stopped at 6/10: three of five Proposal Compliance checks passed after one correction, but Proposal Compliance exception handling [WFT-038] and Proposal Compliance source traceability [WFT-038] remained unsupported.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
build a compliance matrix from a complex request for proposals
02 · Scope before score
Test disclosures
Input disclosure
Synthetic blind-test input pack WFT-038: A proposal team will supply a solicitation containing mandatory, conditional, repeated, and cross-referenced requirements. Source facts: requirements WFT-038-R01–R12; mandatory SOC 2, 99.9% SLA, 30-day implementation, signed pricing form; response lacks SLA remedy and signature. Governing rule card: mandatory status, response location, compliance state, and remediation owner. Apply every supplied rule in its stated order, abstain on incomplete rows, preserve conflicts as exceptions, and expose the source locator and arithmetic or rationale for every decision. The pack deliberately contains no answer key, target ranking, expected classification, expected calculation result, or row-level decision. Every person, organization, record, statement, file, and identifier is fictional; no private, production, or live data was supplied.
Run disclosure
We ran one text-only synthetic benchmark WFT-038 for “build a compliance matrix from a complex request for proposals” in a Codex multi-agent session. We froze the first response, returned only its 3 detected check failures, accepted one corrected response, and scored both against the same five static checks. No external or live action occurred: nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, emailed, called, messaged, executed, or changed outside the fictional text fixtures. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Run bounded synthetic field test WFT-038. Task: build a compliance matrix from a complex request for proposals. Context: A proposal team will supply a solicitation containing mandatory, conditional, repeated, and cross-referenced requirements. Fictional source facts: requirements WFT-038-R01–R12; mandatory SOC 2, 99.9% SLA, 30-day implementation, signed pricing form; response lacks SLA remedy and signature. Governing policy, formula, or rubric: mandatory status, response location, compliance state, and remediation owner. Apply every supplied rule in its stated order, abstain on incomplete rows, preserve conflicts as exceptions, and expose the source locator and arithmetic or rationale for every decision. Produce a RFP compliance matrix, response locator table, and gap register. Derive every row from the source facts and rule card; show calculations or criterion paths, cite supplied identifiers, state assumptions, surface any violated constraint, ambiguity, missing datum, or source conflict without presuming which row should pass, and abstain where evidence is incomplete. Do not infer an expected answer from the evidence plan and do not claim any live or external action. Planned verification after the frozen response: A requirement matrix with page references and an independent completeness review will verify every obligation.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
Frozen first response WFT-038 produced a RFP compliance matrix, response locator table, and gap register for “build a compliance matrix from a complex request for proposals.” Its first artifact row read “WFT-038-R09 | mark SOC 2 and implementation compliant, SLA partial without remedy, and unsigned pricing as a blocker | status: proposed | source: fictional fixture.” A second row named the missing SLA remedy and unsigned pricing form and left the disposition blank. The rule cell mentioned without verifying mandatory status, response location, compliance state, and remediation owner. No message, transaction, system change, or learner outcome occurred. The audit passed Proposal Compliance task fidelity [WFT-038] and Proposal Compliance handoff usability [WFT-038]. It found for Proposal Compliance rule accuracy [WFT-038], the draft mentioned but did not verify mandatory status, response location, compliance state, and remediation owner; for Proposal Compliance exception handling [WFT-038], the draft left the missing SLA remedy and unsigned pricing form without an explicit disposition; for Proposal Compliance source traceability [WFT-038], the draft gave WFT-038-R09 no source locator. Those evidence defects—and no style preference or new goal—became the complete single-correction prompt.Initial score: 4/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Correct only these detected WFT-038 first-draft failures, using no new input or goal: 1) Proposal Compliance rule accuracy [WFT-038] — the draft mentioned but did not verify mandatory status, response location, compliance state, and remediation owner; 2) Proposal Compliance exception handling [WFT-038] — the draft left the missing SLA remedy and unsigned pricing form without an explicit disposition; 3) Proposal Compliance source traceability [WFT-038] — the draft gave WFT-038-R09 no source locator.06 · Corrected output
Corrected final result
Corrected response WFT-038 preserved all supplied identifiers and the central decision: mark SOC 2 and implementation compliant, SLA partial without remedy, and unsigned pricing as a blocker. Its corrected row read “WFT-038-R09 | rule: mandatory status, response location, compliance state, and remediation owner | decision: mark SOC 2 and implementation compliant, SLA partial without remedy, and unsigned pricing as a blocker | static status: 6/10.” It changed only failed dimensions, adding support for Proposal Compliance rule accuracy [WFT-038]. The final audit passed Proposal Compliance task fidelity [WFT-038], Proposal Compliance rule accuracy [WFT-038], and Proposal Compliance handoff usability [WFT-038]. It still lacked Proposal Compliance exception handling [WFT-038] and Proposal Compliance source traceability [WFT-038]; those failures remain visible. The RFP compliance matrix, response locator table, and gap register earned 6/10 from 3 checks; no second repair was attempted. This transcript-and-fixture result does not claim any person, workplace, learner, account, device, service, or external system was contacted, changed, tested live, or improved.Final score: 6/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Proposal Compliance task fidelity [WFT-038] | Pass | Pass | WFT-038 static check 1 inspected “Proposal Compliance task fidelity [WFT-038]” against WFT-038-R09, the rule “mandatory status, response location, compliance state, and remediation owner,” and the saved RFP compliance matrix, response locator table, and gap register. The first transcript passed; the corrected transcript passed. Only fictional text counted, never a live outcome. |
| Proposal Compliance rule accuracy [WFT-038] | Fail | Pass | WFT-038 static check 2 inspected “Proposal Compliance rule accuracy [WFT-038]” against WFT-038-R09, the rule “mandatory status, response location, compliance state, and remediation owner,” and the saved RFP compliance matrix, response locator table, and gap register. The first transcript failed; the corrected transcript passed. Only fictional text counted, never a live outcome. |
| Proposal Compliance exception handling [WFT-038] | Fail | Fail | WFT-038 static check 3 inspected “Proposal Compliance exception handling [WFT-038]” against WFT-038-R09, the rule “mandatory status, response location, compliance state, and remediation owner,” and the saved RFP compliance matrix, response locator table, and gap register. The first transcript failed; the corrected transcript failed. Only fictional text counted, never a live outcome. |
| Proposal Compliance source traceability [WFT-038] | Fail | Fail | WFT-038 static check 4 inspected “Proposal Compliance source traceability [WFT-038]” against WFT-038-R09, the rule “mandatory status, response location, compliance state, and remediation owner,” and the saved RFP compliance matrix, response locator table, and gap register. The first transcript failed; the corrected transcript failed. Only fictional text counted, never a live outcome. |
| Proposal Compliance handoff usability [WFT-038] | Pass | Pass | WFT-038 static check 5 inspected “Proposal Compliance handoff usability [WFT-038]” against WFT-038-R09, the rule “mandatory status, response location, compliance state, and remediation owner,” and the saved RFP compliance matrix, response locator table, and gap register. The first transcript passed; the corrected transcript passed. Only fictional text counted, never a live outcome. |
08 · No cleanup by omission
What worked—and what failed
What worked
- WFT-038 bounded “build a compliance matrix from a complex request for proposals” to disclosed fictional inputs and froze the first response.
- WFT-038 exposed WFT-038-R09—mark SOC 2 and implementation compliant, SLA partial without remedy, and unsigned pricing as a blocker—inside the saved RFP compliance matrix, response locator table, and gap register.
- WFT-038 earned inspectable passes for Proposal Compliance task fidelity [WFT-038] and Proposal Compliance rule accuracy [WFT-038] under the unchanged rubric.
What failed or remained weak
- WFT-038 still lacked saved-text evidence for Proposal Compliance exception handling [WFT-038]; that failure remains published.
- WFT-038 still lacked saved-text evidence for Proposal Compliance source traceability [WFT-038]; that failure remains published.
09 · Inspectable record
Evidence notes
A requirement matrix with page references and an independent completeness review will verify every obligation.
- WFT-038 preserves the exact synthetic prompt, frozen first-response account, failure-only correction, corrected-response account, and five boolean decisions together.
- WFT-038 scores are arithmetic: 2 first-pass checks × 2 = 4/10; 3 final-pass checks × 2 = 6/10.
- WFT-038 evaluated only the text/static portion of the declared evidence plan—A requirement matrix with page references and an independent completeness review will verify every obligation.—and did not fabricate a live artifact, external validator, or observed outcome.
10 · Boundary of the claim
Limitations
- WFT-038 is a synthetic benchmark, so its mixed verdict measures fit to the disclosed fictional Proposal Compliance fixtures rather than effectiveness in a real workplace or learning setting.
- WFT-038 used one text-only Codex multi-agent session whose underlying model identifier was not disclosed; different prompts, models, fixtures, or human reviewers could produce different results.