Use absolute baseline and pilot dates.
// illustrative report structure
The first useful output is a decision.
A Punk pilot report should separate measurements from estimates, opportunity from realized value, and promising candidates from routes that have earned production use.
Not actual pilot data
All brackets and status examples below are placeholders. This page demonstrates the report format; it contains no customer, deployment, benchmark, savings, accuracy, or latency result.
// 01 executive decision
[Workload name] observe-only pilot
Recommendation: [stay observe-only / collect more evidence / shadow one candidate / canary a narrow route / stop].
Separate tests, fixtures, and mock traffic.
State the numerator and denominator.
Name the owner and next review date.
// 02 scope and baseline
Define what was actually tested.
| Field | Report value | Evidence source |
|---|---|---|
| Workload | [actor + request + model/tools + output] | Pilot scope |
| Provider posture | [live / mock / mixed; key source] | Runtime configuration |
| Volume baseline | [runs per period] | Existing logs or labeled estimate |
| Cost baseline | [$ and model pricing date] | Provider billing or Punk live routes |
| Latency baseline | [p50 / p95] | Comparable app or run timings |
| Correctness method | [schema, golden set, human sample] | Workflow owner |
| Side effects | [tools and levels 0–4] | Tool inventory |
// 03 observed opportunity
Show route eligibility, not just a savings total.
| Route | Eligible runs | Economic label | Status / evidence gap |
|---|---|---|---|
| Exact cache | [count] | [ghost / verified / none] | [freshness and subject-scope review] |
| Semantic cache | [count] | [ghost / verified / none] | [confidence and shadow status] |
| Tool cache | [count] | [ghost / verified / none] | [read-only level, TTL, authorization] |
| Plan reuse | [count] | [ghost / verified / none] | [fresh reads retained, tool schemas] |
| Artifact / hybrid | [count] | [projected / verified / none] | [replay, shadow, mismatch, policy] |
| Live / ineligible | [count] | none | [novel, ambiguous, stale, or governed] |
Ghost savings are not realized savings. Observe mode serves the live path and estimates what an eligible alternative might have saved. Projections, synthetic results, and blocked work stay separate too.
// 04 proof and correctness
Economics follow acceptable behavior.
Replay
[candidate, case count, passes, skips, mismatch count and severity, comparison method]
Shadow
[fresh sample count, agreement, high-severity mismatches, schema and policy status]
Human review
[reviewer role, sample size, agreement method, estimated rework, unresolved failure modes]
// 05 economics
Keep every number in its lane.
| Metric | Value | Required context |
|---|---|---|
| Live-equivalent spend | [$] | Baseline method and price date |
| Actual spend | [$] | Live and optimized costs included |
| Verified savedUsd | [$] | Only routes actually served |
| Ghost savedUsd | [$] | Observe estimate, never added to verified |
| Projected savings | [$] | Volume, eligibility, confidence, and time assumptions |
| Latency saved | [ms] | Comparable routes and distribution |
| Human rework | [$ / hours] | Subtract or report beside savings |
// 06 decision register
End with owners and gates.
| Decision | Owner | Required evidence | Target review |
|---|---|---|---|
| [continue observe] | [name / role] | [traffic or identity gap] | [absolute date] |
| [shadow candidate] | [name / role] | [replay threshold and policy review] | [absolute date] |
| [canary route] | [name / role] | [shadow, mismatch, rollback, approval] | [absolute date] |
Make your pilot inspectable.
The report should be useful even when it concludes that a workflow is not ready to optimize.