// evaluation guide
An alternative agent route needs more than a plausible output. Replay checks it against recorded execution; shadow compares it with fresh live traffic while the user still receives the live result. Together, they reduce risk without claiming that every task is fully deterministic.
Punk uses replay and shadow evidence to qualify eligible reusable paths. Evaluation quality depends on the traces, checks, task class, tool declarations, and policy available for your workload; it is not a universal correctness guarantee.
// two complementary modes
Execute a candidate against recorded requests and, where appropriate, recorded tool results or a controlled read environment. Compare structural fields, schemas, semantics, errors, expected cost, and intended side effects.
Run the candidate beside the current live path, record its result, and return the live response to the user. Suppress or dry-run side effects, then evaluate the difference and update confidence.
// evaluation design
| Task class | Useful checks | What not to assume |
|---|---|---|
| Fully deterministic | Exact equality, canonical JSON, known error behavior | A string match alone covers freshness or authorization |
| Schema deterministic | Schema validity, required fields, field-level rules | Valid JSON means the business answer is correct |
| Semantically comparable | Calibrated similarity, rubrics, sampled human review | A judge score should be the sole gate for risky work |
| Tool plan reuse | Tool sequence, argument constraints, fresh read behavior | Recorded tool output remains current |
| Side-effecting work | Dry-run equivalence, policy, approval, idempotency | Replay or shadow may repeat a write |
Side effects are sacred. Replay must never duplicate them. Shadow must suppress or dry-run them. High-impact actions may remain live and approval-gated even after a candidate performs well.
// qualification sequence
Record the inputs, output, identity, policy context, relevant tool behavior, and outcome needed to understand what a correct run means.
Document scope, required inputs, freshness, output schema, side-effect level, known failure modes, and the checks that will reject it.
Use a held-out set where possible. Review structural and high-severity failures rather than reporting only an average score.
Keep the live provider response as the customer-visible result. Compare candidate output, errors, latency, and suppressed intended effects.
Promote only when policy allows and evidence meets the workload’s threshold. Preserve rollback, live fallback, and a route explanation.
Start observe-only to collect the execution history you need, or request a review for a narrow workload with clear quality and side-effect boundaries.