A less dismal science, redux

Last summer DTLR made a rare foray into the behavioral sciences, commenting on a critique of academic economics found in The Economist. Well Bloomberg View has an editorial this week with a more comprehensive critique of academic economics, one that resonates well with DTLR.  Check it out.

Happy New Year

DTLR expects the new year to bring more scrutiny to non-reproducible research, and poor study design, conduct, analysis, and reporting.  Here is an example from last month in Nature.

"To err is human, but so often?" - David Freedman

Nature's editorial this week discusses the unleashing of the "statcheck" computer program on psychology journal articles.  Evidently it is an automated mechanism to detect errors in the calculation of p-values reported in published papers.

While I do not object to anything they said in the editorial, my concern is that there is very little penalty for carelessness in scientific research.  P-values are actually the least of my concerns; of greater concern are errors or even sub-optimal practices in the design, execution, and reporting of research.  Statistical inference is of no value if these other issues are present, and even when not present, statistical inference remains of incredibly limited value compared to a descriptive presentation of the data.  There are several reasons for this, such as:

• Statistical inference presumes some kind of generalization, usually to a larger, stable population of which the data in the study can be thought of as representative.  This is rarely justified.
• The statistical analysis adds information to the data  in the form of an assumed probability model.  This model's assumptions may well influence the outcome more than the data does.
• Statistical inference is an inherently confirmatory activity, while most research is exploratory.  Statistical models in this context are overfitted to the data, and the generalization implied by statistical inference is invalid.

Nonetheless, sloppy calculation is a sign of carelessness, and for this reason the "statcheck" episode has certainly done a service, if it dis-incentivizes future carelessness.  On the other hand, legitimate criticisms of "statcheck's" own error rate have been raised.  I see this as the needed back-and-forth in the ongoing discussion of reproducible research, and accusations of "harassment" on the part of "statcheck's" creators are over-sensitive and unwarranted.

Book review: The Five Ages of the Universe, by F. Adams and G. Laughlin

Adams and Laughlin (1999) propose that the history of the Universe (past and future) be classified into five periods, not unlike geologic eras, based on what we've learned from physics and cosmology so far. These eras include the primordial era, the stelliferous era (in which we now live), the degenerate era, the black hole era, and the dark era. Much of this is an extrapolation of our physics into the far future of the Universe, and is thus somewhat speculative. The authors propose a Copernican time principle, which states that the era in which we humans find ourselves is not a privileged one in the history of the Universe. (Copernicus earlier showed the our location (Earth) is not a special one in the solar system.)

Reference

Fred Adams and Greg Laughlin, 1999: The Five Ages of the Universe: Inside the Physics of Eternity. New York: Free Press.

Book review: The Trouble with Physics, by Lee Smolin

This controversial book (Smolin, 2006) is an attack on string theory and its dominance in theoretical physics. The author has worked in string theory himself, as well as in a major competing approach, loop quantum gravity. He argues that string theory has led nowhere despite being the dominant approach in the field for a long time. His scientific criticisms of string theory are heavily disputed in the community, and he may have overstated the case. However, he also makes a sociological criticism, that (in the U.S.) string theory has suffocated funding and employment opportunities for physicists who pursue alternative theories. Here Smolin's case seems more compelling.

The book has four parts. The first, "The Unfinished Revolution", is an enjoyable capsule history of unification in theoretical physics. Here the author proposes his list of the five great unsolved problems in physics: (1) Combining quantum theory with general relativity, (2) Resolving the difficulties in the foundations of quantum theory, perhaps by replacing it, (3) finding a theory unifying particles and forces, (4) explain the values of the free constants in the Standard Model of Particle Physics, and (5) explain dark matter and dark energy; alternatively explain the values of the constants in the Standard Model of Cosmology. The second part of the book, "A Brief History of String Theory", is precisely that. Here is where Smolin presents his assessment of the successes and alleged failures of string theory. I found this to be the toughest going and least enjoyable part of the book. The third part, "Beyond String Theory", has three chapters. The first discusses experimental and observational anomalies--for me, this was the most exciting part of the book. The other two chapters discuss speculative theories of physics, alternatives to both currently established theory and string theory. The final part of the book, "Learning from Experience", delves into the philosophy and sociology of physics. This is perhaps the most important part of the book. He feels that theoretical physics has run aground and is ripe for a paradigm shift. The "shut up and calculate" mentality that has been successful for the last 60 years has run its course, and it may be time for radical new ideas. The structure and sociology of the physics community is currently an obstacle to any such radicalism. Although I cannot go all the way along with Smolin in this section, I too am a critic of the academic tenure system and the funding mechanisms for science in the U.S.

Smolin has thought a great deal about the history, philosophy, and sociology of the physics profession. This is unusual for a physicist nowadays. His book presents an opportunity for the rest of us to do so too.

Reference

Lee Smolin, 2006: The Trouble with Physics: The Rise of String Theory, the Fall of a Science, and What Comes Next. Houghton Mifflin.

Book review: Calculated Risks, by Gerd Gigerenzer

Gigerenzer (2003) addresses statistical thinking, and the lack thereof, in medical and legal contexts, focusing on handling probabilities (risks). He identifies several issues and correctives:

• The illusion of certainty. For instance, most patients are not told that diagnostic medical tests can make mistakes, and are not informed of the error rates (false positive and false negatives).
• Ignorance of risk. Even if uncertainty is acknowledged, laymen and experts often do not know how great the level of risk is.
• Miscommunication of risk. Because of the peculiarities of human psychology, the way that risk information is usually communicated (using probabilities expressed as frequencies) can be misleading. For instance, absolute risk reduction, relative risk reduction, and number needed to treat are all mathematically equivalent ways to express the efficacy of a treatment. However, relative risk reduction is usually the way to communicate the results that leaves the best impression on the untutored mind.
• Clouded thinking. Even when risks are communicated properly, both experts and laypeople may not know how to reason with them. Expressing probabilities as natural frequencies forces us to focus on the reference class, and it allows people with little training to carry out Bayes Rule calculations easily.

The book provides a number of interesting case studies: the cases of breast cancer screening and AIDS counseling are particularly dramatic, and should be required reading for anyone taking a diagnostic medical test. Several other decision making heuristics that can result in misleading results are discussed, such as the "category effect". In general though, the book is not a complete discussion of the psychology of judgment and decision making applied to statistical thinking. Nonetheless, the author identifies a number of sub-optimal medical and legal practices in every day life.

Reference

Gerd Gigerenzer, 2003: Calculated Risks: How to Know When the Numbers Deceive You. Simon & Schuster.

The Atlantic weighs in on reproducible research

DTLR has dwelled on the reproducibility crisis since its birth.  Most of the discussions I have cited are from the scientific literature, but some have appeared in venues intended for a general audience.  One of the best I've seen has just appeared in the Atlantic, in a post by Ed Yong.  It appropriately places the focus on the incentive system for scientists.  A one sentence summary is provided by its quotation of Richard Horton, editor of the Lancet, who said:  "No one is incentivized to be right.  Instead, scientists are incentivized to be productive."

Please take a look at Yong's post.