Decision science / Playbook
A model is only worth what changes because of it. This is how I take analytics from a notebook to a decision people make differently: four gates, seven stages, and one working sheet for each. Pick a stage.
Method and principle only. Every example links to a live project built on synthetic or public data.
Gate 01 · Quality · Stage 01
What will be decided, by whom, and what changes if the answer changes?
One named decision with an owner, the real alternatives, and the cost of being wrong in each direction. The model is not mentioned yet.
Starting from the data or the algorithm, then looking for a decision that fits the output.
Gate 01 · Quality · Stage 02
What do we actually know, what is missing, and how did the data come to exist?
Every source labeled by provenance and readiness. Selection effects are named: outcomes only exist for the cases that were approved, offered or served.
Treating what was observed as the whole population.
Gate 01 · Quality · Stage 03
What is the simplest model that supports the decision, and what does it beat?
A baseline first. Calibration and error cost reported next to accuracy. Limits written down beside the result.
Optimizing a leaderboard metric that nobody in the decision cares about.
Gate 02 · Economics · Stage 04
How does a score become an action, and where does a person step in?
Thresholds tied to costs and capacity, hard rules kept apart from preferences, and an explicit path for human override with reasons recorded.
Shipping a score and assuming the process around it will sort itself out.
Gate 03 · Trust · Stage 05
Do the people who decide actually use it, and do they trust it for the right reasons?
Adoption measured as changed decisions, not logins. Override and agreement rates read as signals, with reasons.
Performative adoption: the tool is used to look data-driven while the decision is made elsewhere.
Gate 04 · Measurement · Stage 06
Did the decision get better because of the model, or would it have anyway?
A comparison that could have said no: holdout, champion/challenger, or a quasi-experimental design, with a rollout gate agreed in advance.
Reporting before-and-after change and calling it lift.
Gate 04 · Measurement · Stage 07
How will we know it has stopped working, and who is allowed to switch it off?
Monitoring tied to the decision metric, a documented model one-pager, a review cadence, and a stated retirement trigger.
A metric becomes the target and people optimize the metric instead of the outcome (Goodhart’s law).
Self-check
The workbenches
Steps on one side, a sheet you can fill in on the other. It calculates as you type, saves in your browser only, and prints to PDF.
Open conversations