Whether a test can be trusted, what a model is worth, when people should follow it and what happens to a metric once it becomes a target. Each note explains the idea and lets you try it.
Synthetic data, illustrative numbers, a few minutes each.
Most A/B tests are decided before they start: too small to detect the effect that matters, or stopped the day they look good. Both produce confident wins that disappear in production.
Try this: Turn on daily peeking and count the false winners, then switch on CUPED and watch the required sample size fall.
Model quality is the easiest number to improve and often the least valuable. Capacity, cost and adoption usually decide what a model is worth, and they are rarely in the business case.
Try this: Raise AUC, then raise adoption by the same effort, and see which moves net value more.
A recommendation only creates value when a person acts on it. People under-trust good models and over-trust bad ones in predictable ways, and workflow design can correct both.
Try this: Make ten calls with an AI adviser and get your own reliance profile: when you followed it and should have, and when you should not.
Every metric that becomes a target starts to drift from the outcome it was meant to track. The drift is quiet, because the dashboard keeps improving.
Try this: Increase the pressure on the team and watch the reported number and the real outcome part ways.
The pricing copilot uses all four: a properly powered test, value at measured adoption, recommendations people can check, and a success measure that cannot be gamed.
