// evaluation guide
Punk compares a reusable result with past and current work before it can serve. Until the evidence is strong enough, users keep receiving the answer from the original model.
Punk uses replay and shadow evidence to qualify eligible reusable paths. Evaluation quality depends on the traces, checks, task class, tool declarations, and policy available for your workload; it is not a universal correctness guarantee.
// two complementary modes
Execute a candidate against recorded requests and, where appropriate, recorded tool results or a controlled read environment. Compare structural fields, schemas, semantics, errors, expected cost, and intended side effects.
Run the candidate beside the current live path, record its result, and return the live response to the user. Suppress or dry-run side effects, then evaluate the difference and update confidence.
// evaluation design
| Task class | Useful checks | What not to assume |
|---|---|---|
| Fully deterministic | Exact equality, canonical JSON, known error behavior | A string match alone covers freshness or authorization |
| Schema deterministic | Schema validity, required fields, field-level rules | Valid JSON means the business answer is correct |
| Semantically comparable | Calibrated similarity, rubrics, sampled human review | A judge score should be the sole gate for risky work |
| Tool plan reuse | Tool sequence, argument constraints, fresh read behavior | Recorded tool output remains current |
| Side-effecting work | Dry-run equivalence, policy, approval, idempotency | Replay or shadow may repeat a write |
Side effects are sacred. Replay must never duplicate them. Shadow must suppress or dry-run them. High-impact actions may remain live and approval-gated even after a candidate performs well.
// qualification sequence
Record the inputs, output, identity, policy context, relevant tool behavior, and outcome needed to understand what a correct run means.
Document scope, required inputs, freshness, output schema, side-effect level, known failure modes, and the checks that will reject it.
Use a held-out set where possible. Review structural and high-severity failures rather than reporting only an average score.
Keep the live provider response as the customer-visible result. Compare candidate output, errors, latency, and suppressed intended effects.
Promote only when policy allows and evidence meets the workload’s threshold. Preserve rollback, live fallback, and a route explanation.
Bring one narrow workload. Punk will find the repeated work and test it while the original model still supplies the live answer.