A less dismal science

This blog rarely strays into the behavioral sciences, and for good reasons.  Some of these reasons are outlined in an article in last week's The Economist, in a special insert, "The World If".  This particular piece ponders the scenario, What if economists reformed themselves?  One of the criticisms identified in the article is "model mania"; the author writes, "problems arise when they mistake the map for the territory."  Frankly, I think this is a criticism that applies more broadly, to any area of mathematical modeling where the contact between model and reality is very loose or non-existent.  This occurs when mathematical models are not validated by comparison with actual data; the ultimate validation regime is to predict new phenomena or future data, and compare such predictions with experimental or observational data.  Theories of physics are usually test driven in this way, as are engineering models, and many of those in data science.  Such validation is often lacking in both economics and inferential (as opposed to predictive) statistical modeling in general.  The author of the Economist piece recommends that economists repeat the mantra, "My model is a model, not the model."  DTLR advises all other users of mathematical and statistical models to do the same.

For further reading, see The Financial Modelers' Manifesto by Paul Wilmott and Emanuel Derman (2009).

Exploratory or confirmatory?

In last week's issue of Science, outgoing editor Marcia McNutt was interviewed (Shell, 2016) on the occasion of beginning a term as President of the National Academy of Sciences.  I am going to reproduce a lengthy quote from the interview.

At Science, the paradigm is changing.  We're talking about asking authors, 'Is this hypothesis testing or exploratory?'  An exploratory study explores new questions rather than tests an existing hypothesis.  But scientists have felt that they had to disguise an exploratory study as hypothesis testing, and that is totally dishonest.  I have no problem with true exploratory science.  That is what I did most of my career.  But it is important that scientists call it as such and not try to pass it off as something else.  If the result is important and exciting, we want to publish exploratory studies, but at the same time make clear that they are generally statistically underpowered, and need to be reproduced.

Bravo, Dr. McNutt!  DTLR agrees completely with the sentiment here.  It matters because the statistical dressing that accompanies much scientific research is usually only appropriate for confirmatory studies, or those that McNutt calls hypothesis testing, rather than hypothesis finding (exploratory).  It is rare to find the editor of a major scientific journal express this view in such a crisp, precise manner.   DTLR hopes that her successor, and other editors and referees of scientific journals, follow the lead set by McNutt.  DTLR also recommends all readers of this blog to take a look at Tukey (1980).

Reference

Ellen Ruppel Shell, 2016:  Hurdling obstacles:  Meet Marcia McNutt, scientist, administrator, editor, and now National Academy of Sciences president.  Science, vol. 353, pp. 116-119.

John W. Tukey, 1980:  We need both exploratory and confirmatory.  The American Statistician, vol. 34, pp. 23-25.

Randomized clinical trials, defended

Medical blogger Vinay Prasad has posted a vigorous defense of randomized clinical trials, responding to a recent paper in the New England Journal of Medicine.  It's worth a look.

H/T:  In the Pipeline by Derek Lowe (discussion here).

Nature keeps the heat up on reproducible research

This week's issue of Nature has a good article by Monya Baker on a wide-ranging survey of scientists about reproducible research, and a related editorial.  DTLR is most encouraged by the final table in Baker's article, the ratings of factors most likely to improve reproducibility.  "More robust experimental design" received the most combined "likely" and "very likely" ratings.  I think that this is the right answer.  Also highly ranked were "better mentoring/supervision" and "better understanding of statistics".  This latter one is a tough call, as statisticians themselves seem not to have reached a consensus on how to move forward, as evidenced in the extensive Discussion items published along with the American Statistical Association's Statement on Statistical Significance and P-values, posted in early March.

DTLR expresses thanks to Nature for keeping the drums beating on reproducible research.  The issue is very visible right now, and the community should strike while the iron is hot, in terms of reforming the infrastructure of our community (laboratory practices, publication standards, and incentives for grant funding, promotion, and tenure).  Mis-aligned incentives are ultimately the cause of non-reproducibility, though methodological issues (poor study design and execution, inappropriate use of statistical methods, etc.) are key enablers.

The water watchdog

DTLR supports the views of Prof. Marc Edwards, expressed in interviews with the Chronicle of Higher Education (with Steve Kolowich, here) and Science Magazine's Working Life (with Rachel Bernstein, here), regarding the mis-aligned incentives for academic scientists, and other topics.  He is one of the experts worked to "uncover and address the elevated lead levels in drinking water in Flint, Michigan" (as Bernstein wrote).

Gravitational waves and colliding black holes

DTLR has been dormant for nearly a half-year.  However a significant discovery has been reported recently that is worthy of note.  The LIGO (Laser Interferometry Gravitational-Wave Observatory) detected the emission of gravitational radiation from a source event that appears to be the collision of two black holes.  The event marks the dawn of the age of gravitational wave astrophysics.  We are again grateful to be witnesses to the making of physics history.  Congratulations to the LIGO and VIRGO collaborations.

Reference:

B.P. Abbott, et al., 2016:  Observation of gravitational waves from a binary black hole merger.  Physical Review Letters, 116:  061102.

The role of institutions in promoting reproducible research

Nature published a superb commentary yesterday by Glenn Begley, Alistair Buchan, and Ulrich Dirnagl, advocating reforms among institutions to help reduce irreproducible research.  I have little to add except unabashed praise.  DTLR endorses the views expressed in the commentary.