// measured public reference study · july 13, 2026
50 held-out issues. 50 exact plans. Zero treatment model calls.
In this narrow AI agent tool-call reuse benchmark, Punk induced one fully specified github_get_issue plan from traces, qualified it with replay and shadow evidence, and reused it while changing the issue number on every request. All 50 treatment plans were followed by successful live GitHub reads.
Evidence label
This is a Punk-sponsored public reference study—not a customer deployment or organic production traffic. It does not imply affiliation with or endorsement by the vLLM project. It measures one repeated model-planning call, not issue triage, issue resolution, or coding quality.
// held-out result
The live control stayed correct. Punk removed the repeated model-planning call.
Both paths chose the exact required read. Only the control paid a model to make that same choice again.
// paired scorecard
Same 50 held-out issues. Alternating request order.
| Measured field | Live-model control | Punk artifact |
|---|---|---|
| Held-out requests | 50 | 50 |
| Exact read-only plans | 50 | 50 |
| Model requests | 50 | 0 |
| Treatment live GitHub reads | Not executed | 50 succeeded |
| Recorded model cost | $0.06835 | $0 |
| Recorded model cost / 1,000 | $1.367 | $0 |
| Client-observed planning latency p50 | 871.55 ms | 8.80 ms |
| Client-observed planning latency p95 | 1,526.46 ms | 18.49 ms |
| Input tokens | 48,100 | 0 |
Measurement boundary: treatment used 100% fewer model requests and 48,100 fewer model input tokens. Client-observed planning latency was approximately 99.0% lower at p50 and 98.8% lower at p95. The separately verified GitHub GET is not included. Cost is recorded model cost from Punk's pricing table, not a provider invoice or realized customer savings; runtime infrastructure and GitHub access were not priced.
// dates and benchmark traffic
Public source records. Fixed benchmark denominators.
519 matching issues
The locked query covered June 13–July 12, 2026 UTC. Because the protocol selected the first 100 results in ascending creation order, the measured sample spans June 13 at 01:25:50 through June 18 at 11:39:58 UTC. Collected July 13 at 17:10:43 UTC.
150 gateway requests
Generated by Punk on July 13 from 17:10:44 to 17:12:39 UTC: 30 observation, 20 shadow, 50 held-out live-model control, and 50 held-out artifact treatment requests.
Why 150 requests from 100 historical issues?
The 30 observation and 20 shadow issue identifiers ran once. Each of the 50 untouched evaluation identifiers ran twice—once through a separate live-model control app and once through the artifact treatment app. These were measured benchmark requests, not incoming vLLM or customer traffic.
// proof before promotion
The artifact had to earn the route.
Observe 30 live-model plans
Thirty benchmark requests asked the live model to select one read-only tool with an issue number that changed each time.
Replay 20 cases
The induced tool-plan artifact recorded 20 passes and zero failures under Punk's existing replay checks.
Shadow 20 held-out requests
The candidate ran silently alongside benchmark live-model requests using previously unseen issue identifiers: 20 passes, zero failures.
Promote, then evaluate once
After the existing gate became eligible and an operator promoted it, the 50 held-out issues were evaluated without replacement.
// what this means
A narrow result, and a useful one.
What it shows
- A repeated, fully specified tool-selection call can be induced from traces.
- Replay and shadow evidence can gate reuse.
- The treatment can stop invoking a model while its source data is still fetched live.
- The benchmark recorded a route and run record for every held-out request.
What it does not show
- A customer deployment, organic production traffic, or realized savings
- Issue triage or plan discovery from an ambiguous request—the prompt named the tool and supplied owner, repo, and number
- Fallback behavior: the study recorded zero treatment fallbacks, so fallback was not exercised
- Issue resolution, production availability, or performance beyond this run
// public evidence record
The result is tied to a clean commit and a frozen dataset.
| Punk commit | 56df8f7bba87626316c33d1e4e8119c622d4de3e |
|---|---|
| Live provider / model | Anthropic / claude-haiku-4-5-20251001 |
| Dataset SHA-256 | f9a612ebf19dcc6408315552ba15c4f7cc081ead8d55b6494abcc95713a401ce |
| Raw report integrity hash | 654d87cda56e4a7b33d8696ba0d739759796e7883db6fc1bf01f690b815d0f13 |
| Public source | Exact vLLM issue query |
| Public evidence | Protocol · Frozen dataset · Sanitized result |
Integrity boundary: the raw per-request report remains private because it includes internal run identifiers. Its hash is an integrity commitment, not a public reproduction artifact. The public files expose every source field used for the dataset hash, the aggregate result, and the pre-specified method.
// direct answers
Questions a buyer should ask.
Is vLLM a Punk customer?
No. This is an independent Punk-sponsored study using public issue identifiers. It does not imply a customer relationship, affiliation, or endorsement.
Was this production traffic?
No. Punk generated 150 benchmark gateway requests over 100 historical public issue identifiers. A design-partner study is still required for a customer production result.
Could this fixed workflow simply be hard-coded?
Yes. Once known, this exact workflow could be hard-coded. The mechanism under test is whether Punk can induce and qualify the reusable route from traces without a team implementing the optimized route by hand. More variable workflows may not pass the same gate.
Did Punk avoid all model cost?
On the 50 held-out treatment planning requests, recorded model cost was zero because no model was invoked. The full benchmark made 100 live-model calls during observation, shadow, and control. The study does not price Punk runtime infrastructure or GitHub API access, and it does not claim realized customer savings.
Was the GitHub issue data cached?
No issue data was served from the planning artifact. Punk reused only the retrieval plan; all 50 treatment plans were followed by successful GitHub GET requests.
Why does this matter for production agents?
Agents often repeat the same preparatory decision before reading changing data. This result isolates that pattern: reuse the qualified decision and still fetch the underlying source live. Punk is designed to send ineligible work to the live provider, but this zero-fallback run did not exercise that behavior.
Now measure it on a real workflow.
A design-partner result adds what a public reference cannot: production traffic, customer-reviewed quality, runtime cost, and approved attribution.