Completed field testSynthetic benchmark
Computers · Evidence record
Will This API Rate-Limit Plan Hold Under Bursts: One Verified Gap Remained
This completed synthetic Rate-Limit Testing field test asked the session to design a rate-limit test for bursty API traffic, preserved an actual five-row tiered api rate-limit test matrix, and derived 0/10 then 8/10 from task-specific semantic checks after one failure-only correction.
- Exact prompts and outputs
- One correction only
- Synthetic inputs disclosed
01 · The assignment
The task
design a rate-limit test for bursty API traffic
02 · Scope before score
Test disclosures
Input disclosure
All inputs in DARLT-1121 are fictional and appear verbatim in the exact prompt. Hidden scoring answers were not shown to the response generator. No personal, production, customer, learner, or device data was used. Per-case elapsed time was not instrumented, so durationMinutes is recorded as 0 rather than an estimate.
Run disclosure
A Codex multi-agent session generated one text-only first artifact for “design a rate-limit test for bursty API traffic”. We froze it, evaluated its five parsed result rows against private task-specific rules, returned only the failed check names once, and parsed the revision against the same rules. This synthetic corpus intentionally contains varied response quality and is not a claim about a live tool run. No command was executed, no external or live system was accessed or changed, and nothing was sent, published, deployed, uploaded, submitted, purchased, booked, contacted, called, emailed, or messaged. No external, live, or production action occurred. Per-case elapsed time was not instrumented during the batch session.
- Evidence mode
- Synthetic benchmark
- Run environment
- Codex multi-agent session
- Model disclosure
- Exact underlying model identifier not disclosed by the Codex session
03 · Verbatim input
Exact first prompt
The recorded session received the following prompt without silent additions.
Complete a bounded synthetic field test for: design a rate-limit test for bursty API traffic. Focus: Rate-Limit Testing.
Source scenario: The experiment will define synthetic quotas, user tiers, burst patterns, retry behavior, shared keys, and fairness expectations.
Use only these five public fictional inputs; the scoring answers are intentionally withheld:
DARLT-1121-I1: Free tier allows 60 requests per 60-second fixed window per account; Pro allows 120. Window W1 is 10:00:00-10:00:59.
DARLT-1121-I2: Free account F1 sends requests 1-65 at 10:00:10. A rejected response at that time requires status 429, Retry-After 50, and remaining 0.
DARLT-1121-I3: Keys K-A and K-B belong to Free account F2. K-A consumes 40 requests; K-B then attempts 25 in the same window.
DARLT-1121-I4: Pro account P1 and Free accounts F3/F4 send interleaved bursts. Scheduler policy is round-robin across accounts with per-request wait below 200 ms; one limited account must not block another.
DARLT-1121-I5: At 10:01:00 W2 begins. F1, F2, and P1 each send one request; expected remaining counts are 59, 59, and 119.
Return a concrete tiered api rate-limit test matrix with exactly five result rows, assumptions visible, and no claim that a command, message, booking, transaction, teaching session, or live-system change occurred. Evidence target: A load generator and timestamped response log will verify thresholds, headers, retries, isolation, fairness, and recovery after bursts.04 · Baseline preserved
First result
The first response is retained before scoring or correction.
TIERED API RATE-LIMIT TEST MATRIX DARLT-1121 — FIRST FROZEN ARTIFACT
Task: design a rate-limit test for bursty API traffic. Evaluation focus: Rate-Limit Testing. This is a fictional, text-only artifact; it does not report a live action.
DARLT-1121-R1 :: RESULT=TIERS=use one 100-request quota for every account
DARLT-1121-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
DARLT-1121-R2 :: RESULT=BURST=reject request60 and report Retry-After60
DARLT-1121-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
DARLT-1121-R3 :: RESULT=SHARED=give K-B a fresh independent quota60
DARLT-1121-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
DARLT-1121-R4 :: RESULT=FAIRNESS=serve P1 completely before Free accounts
DARLT-1121-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
DARLT-1121-R5 :: RESULT=RESET=keep F1 and F2 blocked after 10:01:00
DARLT-1121-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for DARLT-1121; any failed row remains visible because only one correction pass is allowed.Initial score: 0/10
05 · One pass only
Exact correction prompt
Only this single correction was allowed; there was no second repair pass.
Revise DARLT-1121 once. Preserve every row that passed. Reconsider only the failed semantic checks below using the original fictional inputs; the private expected answer is not supplied:
- Apply the two declared tier quotas: input was “Free tier allows 60 requests per 60-second fixed window per account; Pro allows 120. Window W1 is 10:00:00-10:00:59.”; first response was “TIERS=use one 100-request quota for every account”.
- Verify the Free burst boundary and retry metadata: input was “Free account F1 sends requests 1-65 at 10:00:10. A rejected response at that time requires status 429, Retry-After 50, and remaining 0.”; first response was “BURST=reject request60 and report Retry-After60”.
- Share quota across keys on one account: input was “Keys K-A and K-B belong to Free account F2. K-A consumes 40 requests; K-B then attempts 25 in the same window.”; first response was “SHARED=give K-B a fresh independent quota60”.
- Test tier isolation and fairness: input was “Pro account P1 and Free accounts F3/F4 send interleaved bursts. Scheduler policy is round-robin across accounts with per-request wait below 200 ms; one limited account must not block another.”; first response was “FAIRNESS=serve P1 completely before Free accounts”.
- Verify recovery after the fixed-window reset: input was “At 10:01:00 W2 begins. F1, F2, and P1 each send one request; expected remaining counts are 59, 59, and 119.”; first response was “RESET=keep F1 and F2 blocked after 10:01:00”.
Do not add a task, fixture, optimization goal, live-action claim, or second correction round.06 · Corrected output
Corrected final result
TIERED API RATE-LIMIT TEST MATRIX DARLT-1121 — AFTER ONE FAILURE-ONLY CORRECTION
Task: design a rate-limit test for bursty API traffic. Evaluation focus: Rate-Limit Testing. This is a fictional, text-only artifact; it does not report a live action.
DARLT-1121-R1 :: RESULT=TIERS=Free requests1-60 allowed; Pro requests1-120 allowed; W1 10:00:00-10:00:59
DARLT-1121-R1-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
DARLT-1121-R2 :: RESULT=BURST=F1 requests1-60 pass; 61-65 status429; Retry-After50; remaining0
DARLT-1121-R2-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
DARLT-1121-R3 :: RESULT=SHARED=K-A 40 pass; K-B first20 pass then requests21-25 return429
DARLT-1121-R3-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
DARLT-1121-R4 :: RESULT=FAIRNESS=round-robin F3+F4+P1; wait<200ms each; F3 limit does not block F4/P1
DARLT-1121-R4-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
DARLT-1121-R5 :: RESULT=RESET=10:01:00 F1 pass remain59; F2 pass remain59
DARLT-1121-R5-NOTE :: The proposed technical step is static and bounded; no command output or successful device change is invented.
Artifact boundary: exactly five scored rows were frozen for DARLT-1121; any failed row remains visible because only one correction pass is allowed.Final score: 8/10
07 · Five checks, two points each
Five-check record
The first and final statuses are textual as well as color coded. Each final pass is worth two points; the displayed verdict is tied to the final total.
| Check | First | Final | Evidence |
|---|---|---|---|
| Apply the two declared tier quotas | Fail | Pass | Public fixture: Free tier allows 60 requests per 60-second fixed window per account; Pro allows 120. Window W1 is 10:00:00-10:00:59. Semantic rule: The test matrix must preserve each tier's exact count, time window, and account scope. FIRST returned “TIERS=use one 100-request quota for every account”; the private static semantic key accepts “TIERS=Free requests1-60 allowed; Pro requests1-120 allowed; W1 10:00:00-10:00:59”, so it fails. FINAL returned “TIERS=Free requests1-60 allowed; Pro requests1-120 allowed; W1 10:00:00-10:00:59”, so it passes. No live result was counted. |
| Verify the Free burst boundary and retry metadata | Fail | Pass | Public fixture: Free account F1 sends requests 1-65 at 10:00:10. A rejected response at that time requires status 429, Retry-After 50, and remaining 0. Semantic rule: Exactly sixty requests fit, and the remaining fixed-window duration at 10:00:10 is fifty seconds. FIRST returned “BURST=reject request60 and report Retry-After60”; the private static semantic key accepts “BURST=F1 requests1-60 pass; 61-65 status429; Retry-After50; remaining0”, so it fails. FINAL returned “BURST=F1 requests1-60 pass; 61-65 status429; Retry-After50; remaining0”, so it passes. No live result was counted. |
| Share quota across keys on one account | Fail | Pass | Public fixture: Keys K-A and K-B belong to Free account F2. K-A consumes 40 requests; K-B then attempts 25 in the same window. Semantic rule: Keys on the same declared account share one counter, so only twenty additional requests fit. FIRST returned “SHARED=give K-B a fresh independent quota60”; the private static semantic key accepts “SHARED=K-A 40 pass; K-B first20 pass then requests21-25 return429; account total60” or “SHARED=K-A 40 pass; K-B first20 pass then requests21-25 return429”, so it fails. FINAL returned “SHARED=K-A 40 pass; K-B first20 pass then requests21-25 return429”, so it passes. No live result was counted. |
| Test tier isolation and fairness | Fail | Pass | Public fixture: Pro account P1 and Free accounts F3/F4 send interleaved bursts. Scheduler policy is round-robin across accounts with per-request wait below 200 ms; one limited account must not block another. Semantic rule: The test must check latency fairness, cross-account isolation, and the larger Pro ceiling together. FIRST returned “FAIRNESS=serve P1 completely before Free accounts”; the private static semantic key accepts “FAIRNESS=round-robin F3+F4+P1; wait<200ms each; F3 limit does not block F4/P1; Pro ceiling120” or “FAIRNESS=round-robin F3+F4+P1; wait<200ms each; F3 limit does not block F4/P1”, so it fails. FINAL returned “FAIRNESS=round-robin F3+F4+P1; wait<200ms each; F3 limit does not block F4/P1”, so it passes. No live result was counted. |
| Verify recovery after the fixed-window reset | Fail | Fail | Public fixture: At 10:01:00 W2 begins. F1, F2, and P1 each send one request; expected remaining counts are 59, 59, and 119. Semantic rule: A new fixed window clears prior counters, and each first request consumes one unit from its tier quota. FIRST returned “RESET=keep F1 and F2 blocked after 10:01:00”; the private static semantic key accepts “RESET=10:01:00 F1 pass remain59; F2 pass remain59; P1 pass remain119”, so it fails. FINAL returned “RESET=10:01:00 F1 pass remain59; F2 pass remain59”, so it fails. No live result was counted. |
Initial0/10
Final8/10
Verdictworked
RecommendedYes, for this scope
08 · No cleanup by omission
What worked—and what failed
What worked
- DARLT-1121 preserved the exact public prompt, first artifact, failure-only correction, final artifact, and independently derived semantic check results.
- Apply the two declared tier quotas passed because the parsed final answer matched the private fixture rule rather than merely repeating an input identifier.
- Verify the Free burst boundary and retry metadata also passed its task-specific rule with the final answer left visible.
What failed or remained weak
- Verify recovery after the fixed-window reset still failed after the only permitted correction; its final value and expected semantic rule remain quoted in the evidence.
09 · Inspectable record
Evidence notes
A load generator and timestamped response log will verify thresholds, headers, retries, isolation, fairness, and recovery after bursts.
- DARLT-1121 stores the public five-input fixture separately from the private semantic answer strings quoted only after evaluation.
- DARLT-1121's first and final scores were recomputed from parsed RESULT rows: 0 and 4 passes multiplied by two.
- DARLT-1121 preserves every unresolved final mismatch; the source evidence plan remains unexecuted because this is a static synthetic benchmark: A load generator and timestamped response log will verify thresholds, headers, retries, isolation, fairness, and recovery after bursts.
10 · Boundary of the claim
Limitations
- DARLT-1121 is a static synthetic response benchmark, not evidence that the task succeeded with a real person, organization, device, account, service, or environment.
- DARLT-1121 uses one Codex multi-agent transcript and a private deterministic fixture key; another prompt, model, evaluator, or real-world input could produce a different result.