Shadow traffic as a pre-release gate

Run the candidate model on a copy of every live request, log its output, and serve users only the current model's. It exercises the real request distribution without exposing users to the candidate's answers.

Unit tests verify that code behaves as specified. Offline evaluation verifies that a model scores acceptably on historical data. Neither verifies behaviour under the current production request distribution, with its real timing, concurrency, and the malformed input that arrives once per million requests. Shadow traffic does.

The mechanism is straightforward. At the point where a live request is routed to the production model, a copy is made, tagged as shadow, and dispatched to the candidate. The candidate's output is recorded alongside production's. The client receives only production's response.

Fork location

Fork at the router or the edge, not within the model service. The fork must be asynchronous: enqueue the copy and return immediately. If the candidate is slow or unavailable, the only observable effect should be queue growth. A synchronous fork couples the candidate's latency to the production path, and the candidate's p99 becomes the product's p99.

Copy the full request, including headers, together with the production response and a request identifier. All of it is needed when a discrepancy requires explanation.

Comparison

Output parity applies when the candidate is intended to be equivalent: a new serving path for the same model, a refactor, or a backend migration. The tolerance is zero, and any difference indicates a defect in one of the two paths.

Output distribution applies when the candidate is a new model and differences are expected. Compare the shape: mean, percentiles, and the fraction of requests on which the two disagree by more than a defined threshold. A new pricing model averaging 3 percent higher is expected. One returning zero on 4 percent of requests indicates an input-handling defect that offline evaluation did not encounter.

Latency should be measured at the candidate rather than end to end. The candidate runs on separate hardware under shadow load, so absolute values are not comparable to production; the distribution under the real request mix is.

Error rate and error class. Exceptions, timeouts, and fallback paths taken. A candidate that falls back to a default on 2 percent of inputs is reporting a validation gap the offline data set could not reveal.

Duration

At minimum one full daily cycle, since traffic has a daily shape and defects cluster at peaks. One week if the system has weekly patterns or a nightly refresh that could interact with the change. Statistical sample size is reached within an hour; the purpose of the longer window is coverage of every operating condition.

Limits

Shadow traffic does not measure outcomes. A candidate can match production's output distribution and still be an inferior model, because the user's response to the shadow output is never observed. That requires a controlled experiment with real exposure. Shadow is the gate before the experiment, and its scope is to establish that the candidate is safe to expose, not that it is better.

It also does not exercise writes. If the candidate has side effects, shadow mode must suppress them, and the suppression is itself code that can fail. Keep the candidate read-only in shadow, or direct its writes to a separate store.

Cost and value

The cost is a queue, a comparison job, and compute for a second model instance. In return, the candidate is exercised on the true request distribution with no user exposure, which no other pre-release test provides. Once the infrastructure exists, every subsequent change to the serving path inherits it. It should be built early and made the first step of every rollout.