Enterprise Pilot
Use this path when a business wants to evaluate Punk on real agent traffic without changing production behavior on day one. The goal is to build evidence in order: observe-only traffic, first-week report, shadow pilot, canary, proof review, then expansion.
This is not a separate product surface. It is the operating sequence for using the gateway, dashboard, governance, learning, artifacts, and approvals with low initial risk.
Pilot Shape
Start with one repeated job, one owner, one app identity, and one observe-mode key.
Good first jobs:
| Job | Why it works |
|---|---|
| Support triage | Repeated inputs, reviewable categories, low-risk reads, clear output schema. |
| Vendor review | Evidence collection, spend thresholds, approval gates, reusable scorecards. |
| Pricing monitor | Scheduled web reads, structured extraction, diffs, low write risk. |
| Lead enrichment | CRM and web reads first; writes can be added after approval policy is proven. |
| Compliance precheck | Evidence burden is explicit and high-impact actions can stay human-approved. |
Avoid first jobs that are mostly one-off creative writing, unbounded brainstorming, or high-impact writes. Those can still use Punk later, but they are poor starting points for proving route safety and savings.
Phase 1: Observe-Only Integration
Objective: collect trustworthy traces without changing serving behavior.
- Create a tenant API key in
observemode. - Route one existing OpenAI- or Anthropic-compatible app through Punk.
- Send
X-Punk-App,X-Punk-Agent, andX-Punk-Subjecton every request. - Keep provider keys in Punk's hosted provider configuration or tenant BYOK vault.
- Add SDK tool tracing only for the tools that matter to the first job.
- Classify side effects before any write path is considered eligible for optimize mode.
Observe mode means Punk returns live provider responses while recording the policy decision and the route it would have used. This is the right default for first production-like traffic.
Minimum evidence after the first day:
| Evidence | Where to inspect |
|---|---|
| Runs are visible with route, cost, latency, and trace events. | Runs |
| App, agent, and subject identity are present. | Run detail |
| Policy verdicts are understandable. | Run detail and Governance audit |
| Repeated shapes start to group or explain abstention. | Patterns and Learning |
| Ghost savings are visible when an optimization would have applied. | Overview, run explanation, PunkBar |
Phase 2: First-Week Report
Objective: turn observed traffic into a decision record.
Produce the report after enough representative traffic has run for a few business days. Do not wait for a perfect optimization story; the first report should make clear what Punk can already explain and what needs more evidence.
Include:
| Section | Questions to answer |
|---|---|
| Traffic covered | Which apps, agents, subjects, models, and tools were observed? |
| Cost and latency baseline | What did live provider traffic cost, and where were the slow paths? |
| Repeated work | Which patterns appeared, which did not, and why? |
| Governance | Which actions would have been allowed, blocked, or approval-required? |
| Side effects | Which tools are level 0-4, and which writes remain out of scope? |
| Candidate optimizations | Exact cache, semantic cache, tool cache, model substitution, artifact, or no action. |
| Evidence gaps | Missing identity, unstable prompts, unclassified tools, insufficient samples, unclear output contract. |
| Recommended next step | Stay observe-only, start shadow, add tracing, tighten policy, or stop the pilot. |
Use the report to decide one narrow shadow pilot. A useful report is operational: it names the workflow, risks, evidence, owner, and decision.
Phase 3: Shadow Pilot
Objective: compare candidate optimized routes against live traffic without serving them.
Shadow only the patterns that have a stable input shape, reviewable output contract, and acceptable side-effect posture. Side effects must be suppressed during replay and shadow. User-visible and high-impact writes should stay live plus approval until their policy and rollback path are explicit.
Shadow readiness checklist:
- The pattern represents real work, not test traffic.
- Output correctness can be checked structurally, semantically, or by a reviewer.
- Tool side-effect levels are declared.
- Replay results exist for relevant history.
- Shadow results compare against live traffic without firing duplicate side effects.
- The workflow owner can explain fallback behavior.
Evidence to review:
| Evidence | Required interpretation |
|---|---|
| Replay pass/fail | Did the candidate match historical traces? |
| Shadow pass/fail | Did it match new live traffic while live remained authoritative? |
| Output diff | Are mismatches acceptable, explainable, or blockers? |
| Policy verdict | Would optimize mode allow the route? |
| Route explanation | Is the decision understandable to an operator? |
| Rollback path | Can the candidate be quarantined or retired without app changes? |
Phase 4: Canary
Objective: serve a proven route to limited traffic with explicit monitoring.
Enable canary rollout only after proof review. Keep the scope narrow:
- One app or agent.
- One pattern or artifact.
- One owner.
- One rollback decision path.
- One monitoring window.
Canary review should watch route mix, savings, latency, policy decisions, output diffs, fallback rate, and user feedback. If confidence drops, quarantine the artifact or return the key to observe mode.
Canary does not remove the live provider path. Punk's router should fail open to live provider traffic when an optimized route is unavailable or unsafe.
Phase 5: Proof Review
Objective: make promotion a recorded decision, not an assumption.
Before expanding traffic, the owner and operator should review:
| Topic | Decision |
|---|---|
| Pattern | What traffic is matched, and what traffic is excluded? |
| Artifact or route | What optimized path will serve? |
| Replay | Which historical traces passed or failed? |
| Shadow | Which live comparisons passed or failed? |
| Side effects | What was suppressed, approved, or never eligible? |
| Policy | Which rules allow, block, or require approval? |
| Savings | What cost and latency reduction is measured, not projected? |
| Rollback | Who can quarantine, retire, or switch back to observe? |
Promotion is ready when the decision can be explained from dashboard evidence and reproduced from the trace ledger. If the evidence is weak, keep the route in shadow or observe mode.
Phase 6: Expansion
Objective: add coverage without making the system harder to reason about.
Expand in this order:
- More traffic for the same app and pattern.
- More subjects for the same workflow.
- More read-only tools with declared side-effect levels.
- Reversible writes with idempotency and approval posture.
- Additional apps that share the same governance model.
- Higher-impact writes only after approval, audit, and rollback are exercised.
Each expansion should have its own observed baseline, evidence review, canary, and rollback path. Avoid mixing pilot test traffic with production-like traffic when judging pattern confidence.
Developer Checklist
- Use the base URL swap first.
- Keep the application request shape stable.
- Send identity headers on every request.
- Store provider keys in Punk, not in application code.
- Add SDK tracing for important tools after model traffic is visible.
- Declare side-effect levels for tools.
- Keep write paths observable before making them optimizable.
- Use feedback when a response is wrong or notably correct.
Operator Checklist
- Issue separate observe and optimize keys.
- Scope keys to apps where possible.
- Review provider key source and credential storage.
- Decide retention and redaction before broad traffic.
- Write policy for the first app and agent.
- Require approval for level 3 and 4 actions during the pilot.
- Review audit events daily during the first week.
- Enable canary only after proof review.
Progressive Disclosure
Do not make every advanced feature part of day-one onboarding. Expose features in the order the pilot needs them:
| Stage | Show first | Defer until needed |
|---|---|---|
| Observe-only | Runs, route explanations, cost, latency, identity, policy verdicts. | Artifacts, model substitutions, connector catalog depth, advanced settings. |
| First-week report | Patterns, learning notes, side effects, ghost savings, evidence gaps. | Manual artifact controls, cross-tenant learning, custom policy tuning. |
| Shadow | Replay, shadow comparison, output diffs, suppressed side effects. | Full workflow editing, global optimization knobs. |
| Canary | Route mix, fallback, confidence, rollback, approval state. | Broad automation, high-impact writes, automatic promotion. |
| Expansion | App-level coverage, owner cadence, policy reuse, audit exports. | Organization-wide optimization defaults. |
Progressive disclosure should hide complexity from first-time users without removing capability. Advanced controls remain available to operators and developers when the evidence says they are relevant.
Definition Of Done
The enterprise pilot has a credible starting point when:
- One real app is routed through Punk in observe mode.
- Representative traffic appears in Runs with complete identity.
- The first-week report names baseline cost, latency, repeated patterns, policy posture, side effects, and evidence gaps.
- One candidate route has replay and shadow evidence or a clear reason it is not eligible.
- Canary criteria and rollback are documented before optimize mode serves production-like traffic.
- The expansion plan adds one risk dimension at a time.