On the impossibility of unambiguously selecting the best model for fitting data

We analyze the problem of selecting the model that best describes a given dataset. We focus on the case where the best model is the one with the smallest error, respect to the reference data. To select the best model, we consider two components: (a) an error measure to compare individual data points, and (b) a function that combines the individual errors for all the points. We show that working with the most general definition of consistency, it is impossible to extend individual error measures in a way that provides a unanimous consensus about which is the best model. We also prove that, in the best case, modifying the notion of consistency leads to expressions that are too ill-behaved to be of any practical utility. These results show that selecting the model that best describes a dataset depends heavily on the way one measures the individual errors, even if these measures are consistent.