Software either works or throws an error. A machine learning model does neither - it keeps returning confident predictions that are gradually less correct, because the world it was trained on has moved. Fraud patterns change, customer behaviour shifts, a product line is discontinued, a supplier changes their document format. Nothing errors. Accuracy declines and the system reports itself as healthy.
Two distinct things drift and they need separate monitoring. Data drift is when the inputs change distribution - the average transaction size doubles, a new customer segment appears, a field starts arriving in a different format. Concept drift is when the relationship between inputs and outcome changes: the same features that predicted churn last year no longer do, because the market changed. Data drift is detectable immediately from the inputs alone; concept drift is only visible once you know the outcomes.
That distinction determines what you can monitor in real time. Input distributions can be compared continuously against the training set, and a significant shift is an early warning that costs nothing to compute. Actual accuracy requires ground truth, which arrives on a delay - you learn whether a loan defaulted months later, whether a prediction of churn was right after the churn window closes. Design for that lag: capture predictions with identifiers so outcomes can be joined to them when they arrive, or you will never be able to measure accuracy at all.
Monitor the outputs too, not just the inputs. The distribution of predictions shifting - a classifier that used to flag two per cent of cases now flagging eight - is a strong signal even before you know which flags were correct. So is a shift in confidence: a model becoming systematically less certain usually means it is seeing inputs unlike anything it was trained on.
Retraining should be triggered by evidence rather than by the calendar. A quarterly retrain is better than nothing and is either too frequent for a stable problem or too slow for a volatile one. Set thresholds - on drift metrics, on measured accuracy once ground truth arrives - and retrain when they are crossed. And validate the retrained model against the current period before promoting it, because a model retrained on drifted data can be worse than the one it replaces.
Two things worth having in place before any of this matters. A held-out evaluation set that reflects current reality, refreshed periodically, so you have something honest to measure against. And the ability to roll back to the previous model quickly - versioned, with its training data recorded - because the failure mode of a bad retrain is the same silent degradation as drift, and the fastest remedy is to put back the version that worked.
