Adaptive runtimes create a new engineering obligation. The moment a system can replace live model work with a cache, plan, artifact, or substitute model, evaluation stops being a reporting feature. It becomes part of execution control.
The most common design mistake is treating every form of agreement as interchangeable. A candidate matched ten historical examples, so it is “proven.” It agreed with live output in shadow, so it is “safe.” A model-based evaluator liked the answer, so it can serve.
Those conclusions skip essential state transitions.
Replay, shadow, promotion, and canary are different mechanisms because they answer different questions. Keeping them separate is what makes an adaptive runtime debuggable.
Replay asks about recorded history
Replay reconstructs an eligible historical input and compares the candidate with an accepted recorded output.
That process needs a strict ground-truth contract. A run should not become replay evidence merely because it completed. In Punk, incomplete model output, failed structured-output validation, terminal integrity errors, and unresolved negative feedback disqualify the recorded output from acting as truth.
The candidate then executes without real tool or network access. Recorded tool results may be provided as inputs, but historical side effects are not repeated. Because the artifact language is interpreted and side-effect-free, a replay cannot send the old email, update the old ticket, or repeat the old payment.
Replay can establish:
- agreement with eligible historical examples;
- deterministic output behavior under the recorded inputs;
- schema and structural compatibility;
- failure cases that the candidate cannot yet handle.
It cannot establish current freshness, coverage of unseen requests, or current authorization.
Shadow asks about present behavior
Shadow runs the candidate beside the live path on current traffic. The live provider result remains authoritative, and the candidate result becomes evaluation evidence.
This catches failures that replay cannot. The distribution may have shifted. A source field may have changed shape. A new class of request may resemble an old pattern while violating an assumption. A hybrid artifact’s model-dependent fields may disagree even when its deterministic subset remains stable.
Shadow can establish current agreement within the observed window. It still does not establish authority to serve.
That distinction matters operationally. If the system lets a shadow candidate alter the user-visible result, it is not shadow. If it repeats external writes for comparison, it is not safe evaluation. If it counts the live output as customer acceptance without feedback or a task-specific quality contract, it may be measuring imitation rather than correctness.
Promotion asks whether the evidence is sufficient
Promotion is a gate over evidence and consequence.
The Punk default gate requires replay passes, shadow passes, a bounded shadow failure rate, confidence, and an active pattern. Deterministic task classes require zero replay mismatches. The gate also distinguishes eligibility from automatic authority.
A deterministic level-zero artifact can qualify for automatic promotion under configured policy. A read-only tool-plan artifact may qualify because it emits a plan rather than executing the read. Write-capable plans, hybrid artifacts, and non-allowlisted types require a human decision even when their quality evidence passes.
This is an important property: quality evidence does not erase consequence.
A candidate can be correct and still lack permission. It can be cheap and still be stale. It can match the live model and still violate a current policy boundary.
Canary asks whether serving evidence remains healthy
Pre-production proof cannot represent every live condition. Canary rollout converts a qualified candidate into bounded production evidence.
When enabled, Punk begins at a narrow traffic share and advances through explicit rungs. Each rung earns its own evaluation window rather than inheriting confidence forever from an earlier stage. Failure evidence holds or degrades the rollout, and operators retain rollback and quarantine controls.
The implementation details can vary across runtimes. The contract should not:
- admit only requests inside the qualified scope;
- keep a reference or fallback path available;
- measure live outcomes separately from pre-production proof;
- require new evidence at each expansion;
- make rollback faster than diagnosis.
Hybrid candidates need a higher bar
A hybrid artifact contains a deterministic program plus one or more model-dependent fields. Replay can certify the deterministic subset without making a paid model call, but it cannot claim that the unresolved model field is correct.
That is why hybrid evidence leans more heavily on shadow. The current Punk defaults require at least ten shadow passes and a higher confidence threshold, and hybrid artifacts are never automatically promoted.
This is not because hybrid execution is inherently bad. It is because the runtime must be honest about which part it proved deterministically and which part still depends on live probabilistic behavior.
Evaluations need identity and policy scope
Two requests can have identical text and different authority.
Tenant, application, subject, policy version, tool contract, and freshness context belong in the eligibility boundary. Evidence collected for one private channel or customer account must not silently qualify a route for another. A current policy denial must beat an old performance win.
The route explanation is where this becomes inspectable. It should identify the method that served, why it qualified, the alternatives rejected, the policy result, relevant artifact or cache evidence, and the available fallback.
A useful proof state machine
The compact engineering model is:
candidate → replay → shadow → approval or promotion → canary → stable
Every arrow needs:
- an explicit evidence predicate;
- an immutable or reviewable transition record;
- a consequence boundary;
- a failure destination;
- an operator action that can reverse or stop serving.
If a runtime has “evaluation” but cannot explain those transitions, the evaluator is advisory. That may still be useful. It is not yet an adaptation contract.
Replay is not shadow. Shadow is not promotion. Promotion is not permanent trust. The technical discipline is preserving those distinctions all the way into routing.
Questions readers ask
Can replay prove that a candidate is safe on current traffic?
No. Replay establishes agreement with eligible historical examples. Current data, request shapes, policy, permissions, and upstream behavior may differ, which is why shadow evidence is separate.
Should a passing shadow candidate serve automatically?
Not necessarily. Promotion also depends on the configured evidence threshold, side-effect level, artifact type, policy, approval requirements, and the operator's rollout posture.