# Public GitHub issue-plan reuse benchmark

Status: pre-specified public-workload protocol committed before execution
Sponsor and operator: Punk Technologies
Evidence label: measured public reference study; not a customer deployment

## Question

Can Punk stop paying a live model to select the same read-only issue-retrieval plan while preserving fresh GitHub reads for unseen issues?

This benchmark tests planning reuse only. It does not test issue resolution, code generation, downstream summary quality, or a customer production deployment.

## Public workload

- Source repository: `vllm-project/vllm`
- Source interface: GitHub public Issues API
- Locked collection window: 2026-06-13 00:00:00 UTC through 2026-07-12 23:59:59 UTC
- Locked query: `repo:vllm-project/vllm is:issue created:2026-06-13..2026-07-12`
- Ordering: GitHub `created` ascending
- Population: first 100 results returned by the query
- Split: first 30 observation, next 20 fresh shadow, final 50 held-out evaluation
- Retained source data: factual issue identifiers, URLs, and timestamps only

The split is chronological and fixed before model output is inspected. No issue may move between splits after a run begins.

## Treatment and paired control

The treatment app observes 30 distinct issue requests, runs learning and replay, then receives 20 previously unseen benchmark requests for shadow comparison against the live model. If the artifact does not pass Punk's existing promotion gate, the benchmark stops and reports the failed gate.

After promotion, each of the 50 held-out issues is sent to:

1. A separate control app that has never participated in learning and therefore uses the requested live model.
2. The treatment app, which may use the promoted tool-plan artifact.

Pair order alternates by issue to reduce simple time-order bias. The gateway's periodic learning cadence must be disabled during the run; only the protocol's explicit learning ticks are allowed.

## Fixed task contract

The agent receives `{ owner, repo, number }` and three declared read-only tools. The correct first plan is exactly one `github_get_issue` call with those three arguments copied without transformation. The treatment must then execute that read against GitHub live. No comments, labels, closures, edits, or other writes are permitted.

## Required report fields

- UTC start and end times
- Punk commit SHA and dirty-state flag
- Source query, collection timestamp, population count, selected IDs, and dataset SHA-256
- Requested model and provider posture
- Observation, shadow, and held-out volumes
- Replay and shadow pass/fail counts
- Route mix and model-request count
- Exact tool and argument agreement
- Fresh GitHub read successes and failures
- Input/output tokens, model cost, and avoided model cost
- Client-observed and persisted p50/p95 latency
- Fallbacks, malformed plans, errors, and all exclusions
- Run IDs and whether each treatment run emitted `model.requested`

Provider cost is the request's recorded token usage multiplied by Punk's contemporaneous pricing table. It is not a provider invoice. Paired client latency is the primary latency measure; Punk's saved-latency estimate is secondary.

## Go/no-go standard

The benchmark is publishable as positive planning-reuse evidence only if:

- The artifact passes the existing replay and shadow promotion gate without manual threshold changes.
- All 50 treatment requests use the artifact route.
- All 50 treatment plans exactly match the required tool and arguments.
- All 50 fresh GitHub reads succeed.
- No treatment run emits `model.requested`.
- No external write is attempted.
- All observation, shadow, and control requests use one consistent non-mock provider/model snapshot and emit `model.requested`.
- The report includes all 50 pairs, with zero exclusions, fallbacks, malformed plans, or unreported errors.

If any condition fails, publish the actual denominator and failure. Do not remove or replace failed issues.

## Required disclosure

> This Punk-sponsored public reference study uses public vLLM GitHub issue identifiers. It is not a customer deployment and does not imply affiliation with or endorsement by the vLLM project. The study measures read-only issue-retrieval planning, not issue resolution or coding quality. Results apply only to the disclosed workload, model, configuration, and dates.