Completed field testSynthetic benchmark
Computers · Evidence record
Check a Software Download's Authenticity With AI and Checksums: All Five Semantic Checks Passed
This completed synthetic Software Integrity field test asked the session to verify that a software download is authentic, preserved an actual five-row software download authenticity verdict, and derived 0/10 then 10/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
verify that a software download is authentic
02 · Scope before score
Test disclosures
Input disclosure
All inputs in VSD-8676 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “verify that a software download is authentic”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: verify that a software download is authentic. Focus: Software Integrity.
Source scenario: The experiment will present official release information alongside intact and altered sample downloads in an isolated setting.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
VSD-8676-I1: Requested package is NoteForge 5.2.1 for arm64, filename noteforge-5.2.1-arm64.pkg, size 84,210,944 bytes; x64 package is out of scope.
VSD-8676-I2: Maintainer release file lists SHA-256 c43a77b0 for the arm64 package; downloaded fixture hashes to c43a77b1.
VSD-8676-I3: Static signature report says cryptographic signature valid but signer is Northwind Test Labs; approved publisher certificate subject is NoteForge Software LLC.
VSD-8676-I4: Fixture download URL host is mirror-download.invalid; approved release host is releases.noteforge.example. No redirect chain connects them.
VSD-8676-I5: Policy requires quarantine recommendation when any identity check fails; execution and deletion are outside the static test. Evidence record must include filename, size, both hashes, signer, and host.
Return a concrete software download authenticity verdict with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: Published checksums, signatures, and known file identities will verify each authenticity decision.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
SOFTWARE DOWNLOAD AUTHENTICITY VERDICT VSD-8676 — FIRST FROZEN ARTIFACT
Task: verify that a software download is authentic. Evaluation focus: Software Integrity. This is a fictional, text-only artifact; it does not report a live action.
VSD-8676-R1 :: RESULT=ARTIFACT=accept any NoteForge package with a similar name
VSD-8676-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
VSD-8676-R2 :: RESULT=CHECKSUM=pass because only the final character differs
VSD-8676-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
VSD-8676-R3 :: RESULT=SIGNATURE=pass because the signature is mathematically valid
VSD-8676-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
VSD-8676-R4 :: RESULT=SOURCE=trusted because the page uses HTTPS
VSD-8676-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
VSD-8676-R5 :: RESULT=VERDICT=install it to see whether it works
VSD-8676-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for VSD-8676; any failed row remains visible because only one correction pass is allowed.Initial score: 0/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise VSD-8676 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Pin the intended artifact: input was “Requested package is NoteForge 5.2.1 for arm64, filename noteforge-5.2.1-arm64.pkg, size 84,210,944 bytes; x64 package is out of scope.”; first response was “ARTIFACT=accept any NoteForge package with a similar name”.
- Compare the exact checksum: input was “Maintainer release file lists SHA-256 c43a77b0 for the arm64 package; downloaded fixture hashes to c43a77b1.”; first response was “CHECKSUM=pass because only the final character differs”.
- Evaluate the signature identity: input was “Static signature report says cryptographic signature valid but signer is Northwind Test Labs; approved publisher certificate subject is NoteForge Software LLC.”; first response was “SIGNATURE=pass because the signature is mathematically valid”.
- Use the disclosed source record: input was “Fixture download URL host is mirror-download.invalid; approved release host is releases.noteforge.example. No redirect chain connects them.”; first response was “SOURCE=trusted because the page uses HTTPS”.
- Issue a bounded decision without execution: input was “Policy requires quarantine recommendation when any identity check fails; execution and deletion are outside the static test. Evidence record must include filename, size, both hashes, signer, and host.”; first response was “VERDICT=install it to see whether it works”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
SOFTWARE DOWNLOAD AUTHENTICITY VERDICT VSD-8676 — AFTER ONE FAILURE-ONLY CORRECTION
Task: verify that a software download is authentic. Evaluation focus: Software Integrity. This is a fictional, text-only artifact; it does not report a live action.
VSD-8676-R1 :: RESULT=ARTIFACT=NoteForge5.2.1 arm64; filename exact; size84210944
VSD-8676-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
VSD-8676-R2 :: RESULT=CHECKSUM=download c43a77b1!=published c43a77b0
VSD-8676-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
VSD-8676-R3 :: RESULT=SIGNATURE=math valid; signer Northwind Test Labs mismatch NoteForge Software LLC
VSD-8676-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
VSD-8676-R4 :: RESULT=SOURCE=mirror-download.invalid unapproved; approved releases.noteforge.example
VSD-8676-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
VSD-8676-R5 :: RESULT=VERDICT=do not install; propose quarantine; record filename+size+hashes+signer+host
VSD-8676-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for VSD-8676; any failed row remains visible because only one correction pass is allowed.Final score: 10/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Pin the intended artifact | Fail | Pass | Public fixture: Requested package is NoteForge 5.2.1 for arm64, filename noteforge-5.2.1-arm64.pkg, size 84,210,944 bytes; x64 package is out of scope. Semantic rule: Authenticity starts with matching product, version, architecture, filename, and byte length. FIRST returned “ARTIFACT=accept any NoteForge package with a similar name”; the private static semantic key accepts “ARTIFACT=NoteForge5.2.1 arm64; filename exact; size84210944”, so it fails. FINAL returned “ARTIFACT=NoteForge5.2.1 arm64; filename exact; size84210944”, so it passes. No live result was counted. |
| Compare the exact checksum | Fail | Pass | Public fixture: Maintainer release file lists SHA-256 c43a77b0 for the arm64 package; downloaded fixture hashes to c43a77b1. Semantic rule: Cryptographic hashes require exact equality; a one-character difference is a failure. FIRST returned “CHECKSUM=pass because only the final character differs”; the private static semantic key accepts “CHECKSUM=download c43a77b1!=published c43a77b0; fail” or “CHECKSUM=download c43a77b1!=published c43a77b0”, so it fails. FINAL returned “CHECKSUM=download c43a77b1!=published c43a77b0”, so it passes. No live result was counted. |
| Evaluate the signature identity | Fail | Pass | Public fixture: Static signature report says cryptographic signature valid but signer is Northwind Test Labs; approved publisher certificate subject is NoteForge Software LLC. Semantic rule: A valid signature from the wrong publisher does not authenticate the requested software. FIRST returned “SIGNATURE=pass because the signature is mathematically valid”; the private static semantic key accepts “SIGNATURE=math valid; signer Northwind Test Labs mismatch NoteForge Software LLC; fail identity” or “SIGNATURE=math valid; signer Northwind Test Labs mismatch NoteForge Software LLC”, so it fails. FINAL returned “SIGNATURE=math valid; signer Northwind Test Labs mismatch NoteForge Software LLC”, so it passes. No live result was counted. |
| Use the disclosed source record | Fail | Pass | Public fixture: Fixture download URL host is mirror-download.invalid; approved release host is releases.noteforge.example. No redirect chain connects them. Semantic rule: Transport encryption alone cannot substitute for the declared publisher-controlled origin. FIRST returned “SOURCE=trusted because the page uses HTTPS”; the private static semantic key accepts “SOURCE=mirror-download.invalid unapproved; approved releases.noteforge.example; chain absent” or “SOURCE=mirror-download.invalid unapproved; approved releases.noteforge.example”, so it fails. FINAL returned “SOURCE=mirror-download.invalid unapproved; approved releases.noteforge.example”, so it passes. No live result was counted. |
| Issue a bounded decision without execution | Fail | Pass | Public fixture: Policy requires quarantine recommendation when any identity check fails; execution and deletion are outside the static test. Evidence record must include filename, size, both hashes, signer, and host. Semantic rule: The failed checksum, signer, and source require rejection while preserving a complete non-executing evidence record. FIRST returned “VERDICT=install it to see whether it works”; the private static semantic key accepts “VERDICT=do not install; propose quarantine; record filename+size+hashes+signer+host; execute0” or “VERDICT=do not install; propose quarantine; record filename+size+hashes+signer+host”, so it fails. FINAL returned “VERDICT=do not install; propose quarantine; record filename+size+hashes+signer+host”, so it passes. No live result was counted. |
Initial0/10
Final10/10
Verdictworked
RecommendedYes, for this scope
08 · No cleanup by omission
What worked—and what failed
What worked
- VSD-8676 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Pin the intended artifact passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Compare the exact checksum also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- The first artifact failed Pin the intended artifact; the one permitted correction resolved it, but the initial defect remains published.
09 · Inspectable record
Evidence notes
Published checksums, signatures, and known file identities will verify each authenticity decision.
- VSD-8676 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- VSD-8676's first and final scores were recomputed from parsed RESULT rows: 0 and 5 passes multiplied by two.
- VSD-8676 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: Published checksums, signatures, and known file identities will verify each authenticity decision.
10 · Boundary of the claim
Limitations
- VSD-8676 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- VSD-8676 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.