Evaluating Models — Accuracy Is the Metric That Lies
The most dangerous number in machine learning is accuracy, because it's the one everyone reaches for and the one that most often misleads. A fraud model that's 99.9% accurate by never catching any fraud is useless and common. Choosing the right evaluation metric — and understanding what each one hides — is the difference between a model that looks good and a model that works. This post is about measuring models honestly.
The most dangerous number in ML is accuracy — a 99.9%-accurate fraud model that catches nothing is common. The confusion matrix, precision vs recall (a business decision), F1/ROC, regression metrics, segment-level evaluation (how bias is caught), and why every offline metric is a proxy (Goodhart's law).