// measured public reference study · july 13, 2026
50 held-out issues. 50 exact plans. Zero treatment model calls.
In this narrow AI agent tool-call reuse benchmark, Punk induced one fully specified github_get_issue plan from traces, qualified it with replay and shadow evidence, and reused it while changing the issue number on every request. All 50 treatment plans were followed by successful live GitHub reads.
Evidence label
This is a Punk-sponsored public reference study—not a customer deployment or organic production traffic. It does not imply affiliation with or endorsement by the vLLM project. It measures one repeated model-planning call, not issue triage, issue resolution, or coding quality.
// held-out result
The live control stayed correct. Punk removed the repeated model-planning call.
Both paths chose the exact required read. Only the control paid a model to make that same choice again.
// paired scorecard
Same 50 held-out issues. Alternating request order.
| Measured field | Live-model control | Punk artifact |
|---|---|---|
| Held-out requests | 50 | 50 |
| Exact read-only plans | 50 | 50 |
| Model requests | 50 | 0 |
| Treatment live GitHub reads | Not executed | 50 succeeded |
| Recorded model cost | $0.06835 | $0 |
| Recorded model cost / 1,000 | $1.367 | $0 |
| Client-observed planning latency p50 | 871.55 ms | 8.80 ms |
| Client-observed planning latency p95 | 1,526.46 ms | 18.49 ms |
| Input tokens | 48,100 | 0 |
Measurement boundary: treatment used 100% fewer model requests and 48,100 fewer model input tokens. Client-observed planning latency was approximately 99.0% lower at p50 and 98.8% lower at p95. The separately verified GitHub GET is not included. Cost is recorded model cost from Punk's pricing table, not a provider invoice or realized customer savings; runtime infrastructure and GitHub access were not priced.
// dates and benchmark traffic
Public source records. Fixed benchmark denominators.
519 matching issues
The locked query covered June 13–July 12, 2026 UTC. Because the protocol selected the first 100 results in ascending creation order, the measured sample spans June 13 at 01:25:50 through June 18 at 11:39:58 UTC. Collected July 13 at 17:10:43 UTC.
150 gateway requests
Generated by Punk on July 13 from 17:10:44 to 17:12:39 UTC: 30 observation, 20 shadow, 50 held-out live-model control, and 50 held-out artifact treatment requests.
Why 150 requests from 100 historical issues?
The 30 observation and 20 shadow issue identifiers ran once. Each of the 50 untouched evaluation identifiers ran twice—once through a separate live-model control app and once through the artifact treatment app. These were measured benchmark requests, not incoming vLLM or customer traffic.
// proof before promotion
The artifact had to earn the route.
-
Observe 30 live-model plans
Thirty benchmark requests asked the live model to select one read-only tool with an issue number that changed each time.
-
Replay 20 cases
The induced tool-plan artifact recorded 20 passes and zero failures under Punk's existing replay checks.
-
Shadow 20 held-out requests
The candidate ran silently alongside benchmark live-model requests using previously unseen issue identifiers: 20 passes, zero failures.
-
Promote, then evaluate once
After the existing gate became eligible and an operator promoted it, the 50 held-out issues were evaluated without replacement.
// what this means
A narrow result, and a useful one.
What it shows
- A repeated, fully specified tool-selection call can be induced from traces.
- Replay and shadow evidence can gate reuse.
- The treatment can stop invoking a model while its source data is still fetched live.
- The benchmark recorded a route and run record for every held-out request.
What it does not show
- A customer deployment, organic production traffic, or realized savings
- Issue triage or plan discovery from an ambiguous request—the prompt named the tool and supplied owner, repo, and number
- Fallback behavior: the study recorded zero treatment fallbacks, so fallback was not exercised
- Issue resolution, production availability, or performance beyond this run
// public evidence record
The aggregate result is inspectable. The original run is not fully reproducible from public material.
| Punk commit | 56df8f7bba87626316c33d1e4e8119c622d4de3e — a historical identifier; the repository was private at publication. |
|---|---|
| Live provider / model | Anthropic / claude-haiku-4-5-20251001 |
| Dataset SHA-256 | f9a612ebf19dcc6408315552ba15c4f7cc081ead8d55b6494abcc95713a401ce |
| Raw report integrity hash | 654d87cda56e4a7b33d8696ba0d739759796e7883db6fc1bf01f690b815d0f13 |
| Public source | Exact vLLM issue query |
| Public evidence | Protocol · Frozen dataset · Sanitized result · Manifest · Offline verifier |
| Current mechanism runner | The manifest identifies the public TypeScript runner and command for a new run against a current deployment. It does not recreate the private historical source or July 13 provider environment. |
Verification boundary: download the manifest, evidence files, and verifier to confirm artifact checksums, dataset checksum, and result arithmetic without keys or access to Punk's repository. The raw per-request report, exact runtime source at the historical commit, provider credentials, provider response records, and live July 13 environment are private or unavailable. The raw report hash is an integrity commitment, not public per-request evidence; these files cannot independently reproduce or prove the original provider-backed run. The current public runner improves mechanism reproducibility for new runs without changing that historical limitation.
For an enterprise evaluation, use the current evidence checklist to distinguish implemented runtime evidence from deployment-specific and unverified items.
// direct answers
Questions a buyer should ask.
Is vLLM a Punk customer?
No. This is an independent Punk-sponsored study using public issue identifiers. It does not imply a customer relationship, affiliation, or endorsement.
Was this production traffic?
No. Punk generated 150 benchmark gateway requests over 100 historical public issue identifiers. A design-partner study is still required for a customer production result.
Could this fixed workflow simply be hard-coded?
Yes. Once known, this exact workflow could be hard-coded. The mechanism under test is whether Punk can induce and qualify the reusable route from traces without a team implementing the optimized route by hand. More variable workflows may not pass the same gate.
Did Punk avoid all model cost?
On the 50 held-out treatment planning requests, recorded model cost was zero because no model was invoked. The full benchmark made 100 live-model calls during observation, shadow, and control. The study does not price Punk runtime infrastructure or GitHub API access, and it does not claim realized customer savings.
Was the GitHub issue data cached?
No issue data was served from the planning artifact. Punk reused only the retrieval plan; all 50 treatment plans were followed by successful GitHub GET requests.
Why does this matter for production agents?
Agents often repeat the same preparatory decision before reading changing data. This result isolates that pattern: reuse the qualified decision and still fetch the underlying source live. Punk is designed to send ineligible work to the live provider, but this zero-fallback run did not exercise that behavior.
Now measure it on a real workflow.
A design-partner result adds what a public reference cannot: production traffic, customer-reviewed quality, runtime cost, and approved attribution.