Machine learning processes frequently result in models that perform well on test sets but fail when applied to real-world data. This issue has been documented in scientific and popular press, including reports of pandemic-era models and a Toronto water quality system that incorrectly indicated safe bathing conditions.
Data Quality and Hidden Variables
Misleading data causes the "garbage in, garbage out" phenomenon, where models learn spurious patterns instead of true features. In Covid prediction datasets, overlapping records and mislabellings allowed models to predict class labels without learning useful information. Similarly, hidden variables like body orientation in chest imaging datasets led models to predict posture rather than disease presence.
Spurious Correlations and Benchmarks
Spurious correlations occur when patterns correlate with class labels but have no true relationship to them. The tank problem illustrates this: a neural network identified tanks by recognizing the time of day because all images were taken at the same time. Common computer vision benchmarks like MNIST and CIFAR contain background pixels spuriously correlated with labels, reducing model generality and increasing sensitivity to adversarial attacks.
Labelling Biases
Data labelling errors also degrade model performance. Human-labelled datasets often capture biases or mistakes, contributing to a reproducibility crisis in science. Even small mislabelling rates of a couple of percent can be significant when researchers compete on accuracies in the tenths of a percent.
