Carlo Rovelli's rant about statistical illiteracy

In the Guardian earlier this week, physicist Carlo Rovelli has an op-ed railing about statistical illiteracy, which he says can be fatal in a pandemic.  He opposes early criticisms of COVID-19 epidemiological models "estimating rather than accurately depicting how severe the virus might be."  Aside from that, he never actually makes clear why statistical illiteracy can be fatal.

Let us focus then on other aspects of his argument that children should be "taught the fundamental ideas of probability theory and statistics".  For example, he writes:

We use probabilistic reasoning every day, and most of us have a vague understanding of averages, variability and correlations. But we use them in an approximate fashion, often making errors. Statistics sharpen and refine these notions, giving them a precise definition, allowing us to reliably evaluate, for instance, whether a medicine or a building is dangerous or not.

On the contrary, we should be using probabilistic reasoning and statistical ideas in an approximate fashion.  To pretend that the nominal precision of probabilistic and statistical claims can be taken seriously in any but the most sterile situations is just as dangerous as the statistical illiteracy Rovelli complains about (see Kay & King, 2020; Weisberg, 2014; Derman, 2011; Taleb, 2007; I won't rehearse the case made eloquently by these writers).  See also my recent post.  

The most important statistical contribution to the evalution of medicines, to take Rovelli's example, is in the rigorous sequence of phased clinical trials (described in ICH, 1997), rather than specific methods of data analysis.  Phased clinical trials are designed to learn and confirm knowledge about the efficacy and safety of a proposed drug or biologic in a sequence of designed studies in three phases, culminating in the third phase consisting of at least two randomized, blinded, controlled clinical trials.  It is the features of study design and mutually reinforcing knowledge from all phases that makes decisions about medicines reliable, with data analytical methodology playing only a supporting role.

Rovelli writes:

Without probability and statistics, we would not have anything like the efficacy of modern medicine, quantum mechanics, weather forecasts or sociology. To take a couple of random but significant examples, it was thanks to statistics that we were able to understand that smoking is bad for us, and that asbestos kills.

The first example, the causal assocation of smoking and lung cancer, is a masterpiece of epidemiological reasoning, described well by Freedman (1999, Sec. 8).  Of one of the statistical models used in the research, Freedman writes,

The realism of the model, of course, is open to serious doubt:  patients are not hospitalized at random.This limits the usefulness of confidence intervals and P-values. Scientifically, the strength of the case against smoking rests not so much on the P-values, but more on the size of the effect, on its coherence and on extensive replication both with the original research design and with many other designs. Replication guards against chance capitalization and, at least to some extent, against confounding—if there is some variation in study design.

Freedman concludes:

The strength of the case rests on the size and coherence of the effects, the design of the underlying epidemiologic studies, and on replication in many contexts. Great care was taken to exclude alternative explanations for the findings. Even so, the argument depends on a complex interplay among many lines of evidence. Regression models are peripheral to the enterprise.

I actually agree with Rovelli that probability and statistics have made important contributions to medicine, quantum theory, and weather forecasting.  However to include sociology in this list is questionable.

Statisicians themselves are in intense disageement over how statistical methods such as p-values should be used and interpreted, as illustrated in last year's special issue of The American Statistician (TAS) on "Statistical Inference in the 21st Century:  A World Beyond p < 0.05", and the "Statistics Debate" sponsored by the National Institute of Statistical Sciences, held earlier this month.  How then do we expect to teach children about statisics?  Perhaps the only positive outcome of the TAS special issue is the final paper of the collection, describing a course on statistical thinking "beyond calculations" (Steel et al., 2019).  I would favor a course of this kind rather than technical training on probability theory and statistical methods that Rovelli seems to be advocating.  Rovelli's rice-throwing story would fit very well within such a course, and illustrates how approximate thinking about statistics can be more concrete and intuitive than a bunch of technical mathematics. 

References

Emanuel Derman, 2011:  Models. Behaving. Badly.  Why Confusing Illusion with Reality Can Lead to Disaster, on Wall Street and in Life.   New York:  Free Press.

David Freedman, 1999:  From association to causation:  some remarks on the history of statistics.  Statistical Science, 14 (3):  243-258. 

International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (1997), ICH Harmonised Tripartite Guideline: General Considerations for Clinical Trials, E8.

John Kay and Mervyn King, 2020:  Radical Uncertainty:  Decision-Making Beyond the Numbers.  New York:  W. W. Norton.

E. Ashley Steel, Martin Liermann, and Peter Guttorp, 2019:  Beyond calculations:  a course on statistical thinking.  The American Statistician, 73 (Supplement 1):  392-401.

Nassim Nicholas Taleb, 2007:  The Black Swan:  The Impact of the Highly Improbable.  New York:  Random House.

Herbert I. Weisberg, 2014:  Willful Ignorance:  The Mismeasure of Uncertainty.  Hoboken:  Wiley.