Matching the method to the decision
| Dimension | What it estimates | What it requires |
|---|---|---|
| Forecast | A quantity over a future horizon, by product, location or period. | Clean history with seasonality, stockouts or missing periods recorded. |
| Classification or propensity | The likelihood that a case belongs to an outcome group. | Labelled historical examples, enough of each class, and a defined intervention. |
| Segmentation | Groups of similar cases or customers worth treating differently. | Defined features, a reason to separate groups, and a matching action per group. |
| Anomaly detection | Cases that differ from expected behaviour. | An expectation of normal, monitoring tolerance and a review path. |
| Ranked prioritisation | Which cases to look at first when capacity is limited. | A ranking target tied to investigation capacity and a measured hit rate. |
Start from the intervention, not the model
Name the action, the person who takes it, how much lead time they need, and what a wrong flag costs. A churn score that surfaces after the renewal conversation has no value; a late-shipment alert that reaches a planner after dispatch is a report, not an intervention. The decision defines the horizon, the granularity and the acceptable error.
- Action, owner and lead time written down
- Cost of false positives and false negatives estimated
- Escalation path for cases the model should not decide
Labels are the constraint
Supervised models need a defensible answer to what happened in each historical case, and that answer is often delayed, incomplete or ambiguous. Churn depends on how you define a lapse; late delivery depends on the promise made at booking; fraud depends on confirmed investigations. Inspect label coverage and delay before choosing a model family, and prefer an observable proxy over an invented certainty.
Model families and what each needs
Forecasting estimates a quantity over time and needs clean history with seasonality and missing periods recorded. Classification and propensity estimate the likelihood of an outcome and need labelled examples with enough of each class. Segmentation groups similar cases and needs a definition of what makes them worth separating. Anomaly detection needs an expectation of normal behaviour, not a label.
Evaluation that survives scrutiny
Compare against the baseline the business uses today: the existing rule, the planner’s judgement or a simple seasonal estimate. Evaluate on time-separated data where the outcome is in the future relative to the inputs, and report error by segment rather than as a single average. Say where the model should not be used.
For a classifier, precision at the team's capacity is often the useful number: if the retention team can call 200 accounts a week, measure how many of the top 200 scores were real cancellations. For a forecast, compare error against a seasonal naive baseline, which repeats last year's value for the same period.
- Baseline established before modelling
- Time-based evaluation with no future information leaking in
- Error reported by segment, horizon and case type
Uncertainty and human judgement
Expose intervals and confidence instead of a single number, and keep a human decision in the path when the consequence is material. Record overrides and their outcomes so the process improves, and define what happens when the model is unavailable, unsure or contradicted by the person closest to the case.
Illustrative example · hypothetical organisation
Worked example: a late-shipment alert
Take a hypothetical distributor that wants to warn customers before a delivery runs late.
The intervention is a proactive message from customer service, which needs at least a day's notice to be useful. So the model scores shipments at dispatch, using only information known at that moment: carrier, lane, warehouse, day of week, order size and recent carrier delays.
The label is late against the promised date, available a few days after dispatch. The baseline is the current rule: flag every shipment on a lane whose recent late rate is above a threshold. The model ships only if, at the volume customer service can handle, it catches more real late deliveries than that rule.
In production, customer-service overrides are logged, actual delivery dates are joined back each week, and performance is reported by carrier, so a drop on one lane is visible.
Operating the model in production
A model is a production system: version the training data, features, code and evaluation set together; promote through review; keep the previous version for rollback; monitor input drift and decision outcomes; and re-evaluate when upstream data or populations move. Without that discipline, quality degrades quietly and the first signal is a business decision going wrong.
Questions and trade-offs
Question
How accurate does a model need to be?
Accurate enough to improve the decision it supports, measured against the baseline the business already uses. That threshold depends on the cost of being wrong and the capacity to act: a low-precision alert stream that overwhelms a team is worse than no model, even if its accuracy looks better on paper.
Question
Can we use machine learning for pricing?
It can estimate how demand responds to price, but the result describes relationships in historical data that were produced by earlier decisions. Treat pricing as an experiment with guardrails for margin, service level and customer impact, and if a controlled test is possible, prefer it to a model that only explains history.
Question
Do we need a data scientist to start?
Not always. Many first predictive use cases start with a rule or a simple model built by an analyst against a clear baseline. A data scientist becomes necessary when label construction, evaluation design or modelling choices outgrow the rule, and when the model needs monitoring in production.
