What to monitor, and who owns it
| Dimension | Why it matters | Owner |
|---|---|---|
| Input and feature drift | The model is being asked about a population it was not built for. | Data engineering with the model owner |
| Prediction distribution | Scores collapse or cluster, which often precedes a decision problem. | Model owner |
| Decision outcomes | The decision the model supports stops producing the expected result. | Business owner of the decision |
| Upstream data quality and freshness | Silent staleness is the most common cause of bad scores. | Data operations |
| Latency and cost per decision | Quality that cannot be afforded at volume is not production-ready. | Model owner with platform operations |
| Evaluation regression | A retrain or prompt change quietly reduces quality. | Model owner, gated at release |
What changes when a model reaches production
A pipeline that fails visibly is easy to operate. A model can degrade silently: inputs shift, populations change, an upstream definition moves, and the output stays plausible while becoming less useful. Production therefore requires evidence of behaviour, not only uptime — which is why evaluation and monitoring have to be designed alongside the model.
A release path with real gates
Promotion should be a reviewed, reversible step: a candidate builds in a lower environment, passes the agreed evaluation set, is compared with the running version on the same cases, and only then receives a share of production traffic. Shadows and canaries catch the failures that offline evaluation cannot, and the previous version stays available for rollback.
- Evaluation gate agreed before the model is built
- Candidate compared with the incumbent on identical cases
- Rollback rehearsed, with the previous version retained
Validate data and features, not only predictions
Most production regressions arrive through data: a feature stops being populated, a join changes cardinality, a source shifts time zone, or training and serving compute a feature differently. Validate input schemas, ranges and distributions before scoring, and keep one definition of each feature used by both training and serving.
- Schema and range checks before scoring
- One feature definition shared by training and serving
- Missing or defaulted inputs surfaced, not silently imputed
Monitoring: drift, outcomes and cost
Four signals usually matter more than infrastructure metrics: whether inputs have moved away from training conditions, whether scores have shifted, whether the decisions taken are producing the expected outcome, and what the system costs per decision. Outcome monitoring needs a defined lag and a comparison group, because an outcome observed later cannot confirm a decision made today.
Drift is measured by comparing the distribution of each input in a recent window with the training data, using a statistic such as the population stability index or a Kolmogorov-Smirnov test, with thresholds agreed per feature. A drift alert is a prompt to investigate, not an automatic retrain: the cause may be a broken upstream join, not a changed world.
- Input and prediction drift with thresholds and owners
- Decision outcomes tracked with an agreed lag and baseline
- Cost per decision measured alongside quality
Rollback and incident ownership
An incident needs a named owner, a decision rule for reverting, and a path that does not depend on the person who built the model being awake. Document what a revert restores — model version, feature set, thresholds — and what it does not. After an incident, record the cause and the control that would have caught it earlier.
- Named owner and escalation path per model
- Revert criteria written before launch
- Post-incident control recorded in the release checklist
Illustrative example · hypothetical organisation
Worked example: releasing a new demand model
Take a hypothetical retailer replacing its store demand model.
The candidate is trained on a fixed snapshot, with the data version, feature code, configuration and evaluation set recorded together. It has to beat the running model on the same backtest weeks, by store size band, before it can be promoted.
It then runs in shadow for two ordering cycles, producing forecasts that are logged but not used. When those hold up against actual sales, it takes over a first group of stores. The previous model stays deployed, and the revert criteria, such as error above an agreed level for two consecutive weeks in any region, are written down before launch.
Environments and reproducibility
Reproducing a result requires the same data, features, code, configuration and evaluation set. Version all five together, record which version produced which decision, and keep enough history to re-run an evaluation that a regulator, an auditor or your own team asks about months later. Reproducibility is also the fastest way to debug a complaint.
Questions and trade-offs
Question
Is MLOps different from DevOps?
It extends it. Versioning, review, automated release and rollback carry over; MLOps adds data and feature validation, evaluation gates, drift and outcome monitoring, and the fact that a passing test suite does not tell you whether the model is still useful. The operating model matters more than the tooling choice.
Question
How often should we retrain a model?
When the evidence says the current version is degrading, not on a fixed calendar. Set drift and outcome thresholds, watch them against a baseline, and retrain through the same evaluation gate as any other release. Scheduled retraining without a monitored reason adds cost and risk without adding quality.
Question
Which MLOps tools do we need?
Fewer than vendor lists suggest: experiment tracking and a model registry (MLflow is a common open-source choice), a way to version data and features, an automated release pipeline, and monitoring that reaches an owner. Pick tools that fit the platform you already operate.
