Take probability models seriously, but not literally

Last week's Science had a feature article about several nurses who were convicted of murder in Europe based on statistical arguments regarding the number of babies who died under their care.  Some of these arguments appear to have been decisively debunked by statisticians not originally involved in the cases.  There is a sidebar article by the same author (Cathleen O'Grady) on mothers who were tried for the murders of their own children, again based on questionable statistical arguments.

These episodes are striking, given that I am currently reading a book by physicist Erica Thompson, Escape from Model Land:  How Mathematical Models Can Lead Us Astray and What We Can Do About It (Basic Books, 2022).  She argues that mathematical models always make simplifications and assumptions that divorce them from the real world; and while the modeling exercise may be of great instructive value, one must often be very careful not to claim that the model output represents reality.  In other words, once you step into "model land", and do your modeling exercise, you must not forget to step back out into the real world, and not mistake model land for reality.  The way I would put it is that models deserve to be taken seriously, but not literally.  In the case of the shoddy statistics that put innocent nurses and mothers in prison, those analyses should not be taken either seriously nor literally. 

Escape From Model-Land

Earlier this week, Mark Buchanan's column in Nature Physics featured "The Limits of a Model" , concerning  the pitfalls of epidemiological forecasting, of course a topic very much in the news in the last few months.  Near the end Buchanan cites an intriguing paper by Thompson & Smith (2019), "Escape From Model-Land", a bracing discussion of why one should be very humble about using mathematical/empirical models.  It calls to mind another entertaining piece from over a decade ago, the "Financial Modeler's Manifesto" by Emanuel Derman and Paul Wilmott.

DTLR strongly recommends the Thompson & Smith piece, a candidate for required reading by every applied mathematician, applied statistician, and mathematical/computational modeler in every discipline.

Reference

Erica L. Thompson and Leonard A. Smith, 2019:  Escape From Model-Land.  Economics:  The Open-Access, Open-Assessment E-Journal, 13:  2019-40.

A less dismal science

This blog rarely strays into the behavioral sciences, and for good reasons.  Some of these reasons are outlined in an article in last week's The Economist, in a special insert, "The World If".  This particular piece ponders the scenario, What if economists reformed themselves?  One of the criticisms identified in the article is "model mania"; the author writes, "problems arise when they mistake the map for the territory."  Frankly, I think this is a criticism that applies more broadly, to any area of mathematical modeling where the contact between model and reality is very loose or non-existent.  This occurs when mathematical models are not validated by comparison with actual data; the ultimate validation regime is to predict new phenomena or future data, and compare such predictions with experimental or observational data.  Theories of physics are usually test driven in this way, as are engineering models, and many of those in data science.  Such validation is often lacking in both economics and inferential (as opposed to predictive) statistical modeling in general.  The author of the Economist piece recommends that economists repeat the mantra, "My model is a model, not the model."  DTLR advises all other users of mathematical and statistical models to do the same.

For further reading, see The Financial Modelers' Manifesto by Paul Wilmott and Emanuel Derman (2009).