A less dismal science

This blog rarely strays into the behavioral sciences, and for good reasons.  Some of these reasons are outlined in an article in last week's The Economist, in a special insert, "The World If".  This particular piece ponders the scenario, What if economists reformed themselves?  One of the criticisms identified in the article is "model mania"; the author writes, "problems arise when they mistake the map for the territory."  Frankly, I think this is a criticism that applies more broadly, to any area of mathematical modeling where the contact between model and reality is very loose or non-existent.  This occurs when mathematical models are not validated by comparison with actual data; the ultimate validation regime is to predict new phenomena or future data, and compare such predictions with experimental or observational data.  Theories of physics are usually test driven in this way, as are engineering models, and many of those in data science.  Such validation is often lacking in both economics and inferential (as opposed to predictive) statistical modeling in general.  The author of the Economist piece recommends that economists repeat the mantra, "My model is a model, not the model."  DTLR advises all other users of mathematical and statistical models to do the same.

For further reading, see The Financial Modelers' Manifesto by Paul Wilmott and Emanuel Derman (2009).

Exploratory or confirmatory?

In last week's issue of Science, outgoing editor Marcia McNutt was interviewed (Shell, 2016) on the occasion of beginning a term as President of the National Academy of Sciences.  I am going to reproduce a lengthy quote from the interview.

At Science, the paradigm is changing.  We're talking about asking authors, 'Is this hypothesis testing or exploratory?'  An exploratory study explores new questions rather than tests an existing hypothesis.  But scientists have felt that they had to disguise an exploratory study as hypothesis testing, and that is totally dishonest.  I have no problem with true exploratory science.  That is what I did most of my career.  But it is important that scientists call it as such and not try to pass it off as something else.  If the result is important and exciting, we want to publish exploratory studies, but at the same time make clear that they are generally statistically underpowered, and need to be reproduced.

Bravo, Dr. McNutt!  DTLR agrees completely with the sentiment here.  It matters because the statistical dressing that accompanies much scientific research is usually only appropriate for confirmatory studies, or those that McNutt calls hypothesis testing, rather than hypothesis finding (exploratory).  It is rare to find the editor of a major scientific journal express this view in such a crisp, precise manner.   DTLR hopes that her successor, and other editors and referees of scientific journals, follow the lead set by McNutt.  DTLR also recommends all readers of this blog to take a look at Tukey (1980).

Reference

Ellen Ruppel Shell, 2016:  Hurdling obstacles:  Meet Marcia McNutt, scientist, administrator, editor, and now National Academy of Sciences president.  Science, vol. 353, pp. 116-119.

John W. Tukey, 1980:  We need both exploratory and confirmatory.  The American Statistician, vol. 34, pp. 23-25.