In scope first
- Read-only classification
- A stable output schema
- A pinned support application identity
- Pseudonymous account, workspace, or queue scope
- Human review of a representative sample
// reference workload · not a customer result
Support triage is a practical first test.
The request shape repeats, output is structured and reviewable, and classification can remain read-only. That makes it a useful example of how an adaptive runtime earns trust.
Evidence label
This is a synthetic/reference workload. It explains pilot design and the evidence a team should collect. It does not claim a customer deployment, measured savings, improved accuracy, or production optimization.
// workload boundary
Classify. Do not mutate.
An agent receives a support ticket and returns fields such as category, priority, and a short internal routing note.
Out of scope first
- Sending a customer reply
- Changing ticket state
- Issuing refunds or credits
- Triggering an escalation automatically
- Mixing incompatible ticket schemas
// candidate routes
Different repetition earns different reuse.
| Route | When it might fit | Evidence before serving |
|---|---|---|
| Observe only | Default during initial measurement | Complete identities, route explanations, and a live-provider baseline |
| Exact cache | An identical request is safe within the same freshness and subject scope | Cache-key review, freshness policy, route explanation |
| Artifact classifier | Category or priority behavior is stable for a narrow segment | Replay cases, shadow cases, mismatch review, policy allowance |
| Hybrid artifact | Deterministic fields are stable but a routing note still needs model judgment | Field-level comparison, fallback behavior, complete cost attribution |
| Live fallback | The ticket is novel, ambiguous, stale, or policy-sensitive | Fallback reason in the route explanation |
// evidence packet
What the pilot must show.
Does it repeat?
Request shapes, schema consistency, relevant segments, sample counts, and exclusions.
Does it still match?
Replay and shadow passes, skipped cases, mismatches by severity, and human-review agreement.
Can it fail safely?
Policy status, side-effect inventory, fallback events, route explanations, canary scope, and rollback.
// metrics · intentionally unfilled
No demo number becomes a customer claim.
A credible publication fills this table from an approved live pilot or labels an internal benchmark as synthetic. Placeholders are not proof.
| Metric | Required value | Source |
|---|---|---|
| Measurement window | [absolute start and end dates] | Pilot plan |
| Tickets observed | [count; tests separated] | Punk runs |
| Stable patterns | [count and eligibility rule] | Pattern report |
| Replay / shadow cases | [counts, passes, skips] | Evidence records |
| High-severity mismatches | [count] | Mismatch review |
| Verified savings | [$ from routes actually served] | Gateway and provider data |
| Ghost savings | [$ estimate; kept separate] | Observe-mode explanations |
| Human review agreement | [% and sample size] | Reviewer-approved sample |
// decision rules
The right result may be “keep it live.”
Too little repetition
Keep observe-only and use Punk for trace visibility and governance. Do not manufacture an optimization story.
Stable structure, judgment-heavy values
Continue shadow comparison and human review before allowing any candidate to serve.
Narrow deterministic segment
Canary only the proven segment while novel and ambiguous tickets continue to the live model.
New write actions
Re-scope. Add side-effect levels, idempotency, approval policy, dry-run handling, and rollback evidence before considering optimization.
Replace the placeholders with your evidence.
Bring a repeated read-only workload and a reviewer who knows what “correct” means.