Monitor the decision, and retire the model on purpose
Models rarely fail loudly. The environment moves, the metric keeps looking fine for a while, and the decision quietly gets worse. Monitoring is a plan for noticing, and a trigger for stopping.
Monitor the outcome the brief named
Track the decision metric from the decision brief, not only model accuracy. Accuracy can hold while the decision gets worse, and the reverse.
Watch inputs, scores and overrides too
Shifts in inputs and score distribution are early warnings; a rise in overrides is the people closest to the work telling you something.
Set the alert line and the retirement trigger now
An alert says look closer; a trigger says stop. Write both as numbers while nobody is attached to the result.
Choose how often to look
The gap between the start of a decline and the check that catches it is lost performance. Look at the cadence of the decision, not the convenience of the dashboard.
Check the metric itself
Once a measure is a target it starts to separate from the outcome. Pair it with a sample reviewed by hand and a second measure that is harder to move.
Retire on purpose
When the trigger fires, switch the model off or replace it, record why, and archive the one-pager. Retirement is an outcome of the process, not a failure of it.
Checklist
Common failure modes
- Monitoring accuracy on stale labels while the decision drifts.
- Alerts nobody owns.
- No retirement trigger, so the model runs until someone remembers it.
See it applied
Open conversations