A Perspective on the Legacy of Edward Lorenz

Recently I stumbled upon a paper published last year by James McWilliams, "A Perspective on the Legacy of Edward Lorenz".  The paper discusses what McWilliams considers to be Lorenz's four most important ideas:

1. Available potential energy in the atmospheric general circulation;
2. The Lorenz model's chaotic dynamics and strange attractor;
3. The limits of weather predictability; and
4. The "slow manifold".
   

My own thesis research directly applies versions of the first two of these ideas, so I owe a lot to him.  I had the privilege of meeting Prof. Lorenz when he was well in his late 80s.  The occasion was an annual meeting of the American Meterological Society in San Diego, in January 2004.  A symposium in Lorenz's honor was held, and he gave the final presentation of the symposium.  Earlier in the day, a poster session was included in the festivities, and he graciously visited the posters.  My mentor and I first encountered him there.  We introduced ourselves and shook his hand, but otherwise exchanged very few words with him.  We also spoke briefly with him after his presentation at the end of the symposium.  One of his biographers described him as "reticent" and that is consistent with my brief experience with him.

Not being part of an elite institution, I do not often meet major figures in the history of science.  My brief encounter with Lorenz was one such instance.  Here is a picture of the cover of the symposium booklet.

Reference

J. C. McWilliams, 2019:  A perspective on the legacy of Edward Lorenz.  Earth and Space Science, 6 (3):  336-350.

Carlo Rovelli's rant about statistical illiteracy

In the Guardian earlier this week, physicist Carlo Rovelli has an op-ed railing about statistical illiteracy, which he says can be fatal in a pandemic.  He opposes early criticisms of COVID-19 epidemiological models "estimating rather than accurately depicting how severe the virus might be."  Aside from that, he never actually makes clear why statistical illiteracy can be fatal.

Let us focus then on other aspects of his argument that children should be "taught the fundamental ideas of probability theory and statistics".  For example, he writes:

We use probabilistic reasoning every day, and most of us have a vague understanding of averages, variability and correlations. But we use them in an approximate fashion, often making errors. Statistics sharpen and refine these notions, giving them a precise definition, allowing us to reliably evaluate, for instance, whether a medicine or a building is dangerous or not.

On the contrary, we should be using probabilistic reasoning and statistical ideas in an approximate fashion.  To pretend that the nominal precision of probabilistic and statistical claims can be taken seriously in any but the most sterile situations is just as dangerous as the statistical illiteracy Rovelli complains about (see Kay & King, 2020; Weisberg, 2014; Derman, 2011; Taleb, 2007; I won't rehearse the case made eloquently by these writers).  See also my recent post.  

The most important statistical contribution to the evalution of medicines, to take Rovelli's example, is in the rigorous sequence of phased clinical trials (described in ICH, 1997), rather than specific methods of data analysis.  Phased clinical trials are designed to learn and confirm knowledge about the efficacy and safety of a proposed drug or biologic in a sequence of designed studies in three phases, culminating in the third phase consisting of at least two randomized, blinded, controlled clinical trials.  It is the features of study design and mutually reinforcing knowledge from all phases that makes decisions about medicines reliable, with data analytical methodology playing only a supporting role.

Rovelli writes:

Without probability and statistics, we would not have anything like the efficacy of modern medicine, quantum mechanics, weather forecasts or sociology. To take a couple of random but significant examples, it was thanks to statistics that we were able to understand that smoking is bad for us, and that asbestos kills.

The first example, the causal assocation of smoking and lung cancer, is a masterpiece of epidemiological reasoning, described well by Freedman (1999, Sec. 8).  Of one of the statistical models used in the research, Freedman writes,

The realism of the model, of course, is open to serious doubt:  patients are not hospitalized at random.This limits the usefulness of confidence intervals and P-values. Scientifically, the strength of the case against smoking rests not so much on the P-values, but more on the size of the effect, on its coherence and on extensive replication both with the original research design and with many other designs. Replication guards against chance capitalization and, at least to some extent, against confounding—if there is some variation in study design.

Freedman concludes:

The strength of the case rests on the size and coherence of the effects, the design of the underlying epidemiologic studies, and on replication in many contexts. Great care was taken to exclude alternative explanations for the findings. Even so, the argument depends on a complex interplay among many lines of evidence. Regression models are peripheral to the enterprise.

I actually agree with Rovelli that probability and statistics have made important contributions to medicine, quantum theory, and weather forecasting.  However to include sociology in this list is questionable.

Statisicians themselves are in intense disageement over how statistical methods such as p-values should be used and interpreted, as illustrated in last year's special issue of The American Statistician (TAS) on "Statistical Inference in the 21st Century:  A World Beyond p < 0.05", and the "Statistics Debate" sponsored by the National Institute of Statistical Sciences, held earlier this month.  How then do we expect to teach children about statisics?  Perhaps the only positive outcome of the TAS special issue is the final paper of the collection, describing a course on statistical thinking "beyond calculations" (Steel et al., 2019).  I would favor a course of this kind rather than technical training on probability theory and statistical methods that Rovelli seems to be advocating.  Rovelli's rice-throwing story would fit very well within such a course, and illustrates how approximate thinking about statistics can be more concrete and intuitive than a bunch of technical mathematics. 

References

Emanuel Derman, 2011:  Models. Behaving. Badly.  Why Confusing Illusion with Reality Can Lead to Disaster, on Wall Street and in Life.   New York:  Free Press.

David Freedman, 1999:  From association to causation:  some remarks on the history of statistics.  Statistical Science, 14 (3):  243-258. 

International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (1997), ICH Harmonised Tripartite Guideline: General Considerations for Clinical Trials, E8.

John Kay and Mervyn King, 2020:  Radical Uncertainty:  Decision-Making Beyond the Numbers.  New York:  W. W. Norton.

E. Ashley Steel, Martin Liermann, and Peter Guttorp, 2019:  Beyond calculations:  a course on statistical thinking.  The American Statistician, 73 (Supplement 1):  392-401.

Nassim Nicholas Taleb, 2007:  The Black Swan:  The Impact of the Highly Improbable.  New York:  Random House.

Herbert I. Weisberg, 2014:  Willful Ignorance:  The Mismeasure of Uncertainty.  Hoboken:  Wiley.

The Nobel Prizes 2014-2020

In its first year, this blog commented here and here on the 2013 Nobel Prize in Physics; however I failed to comment on any of the prizes in following years.  As we are now wrapping up Nobel week 2020, let me offer a few observations on the prizes in physics and chemistry in the years since 2013.  Before I do that, I want to also note last year's Economics Prize for work applying randomized trials to social science.  Such a worthy achievement reflects well on two earlier advocates of randomized trials, statisticians Ronald A. Fisher and Austin Bradford Hill, who pioneered their use in agriculture and medicine, respectively.

Physics

 

I am especially pleased to see two of the subsequent prizes in physics were awarded for technological innovations, namely 2014 (blue LEDs) and 2018 (inventions in laser physics).  It is my view that applied physics is part of physics proper, and should be recognized as such at the highest levels.  Other recent prizes in this vein include 2009 (fiber optics and CCD sensors), half of the 2005 prize (optical frequency comb), and 2000 (the integrated circuit, and semiconductor heterostructures).

 

The 2017 prize to the prime movers of the LIGO collaboration for discovering gravitational waves was arguably premature at the time, as I had misgivings about certain methodological details of the original discovery.  However I think the evidence base accumulated since then, and the methodological improvements, have been both impressive and convincing.  I am prepared to join other voices proclaiming that we are living in the era of gravitational wave astronomy.

The 2019 and 2020 prizes are unusual in that they have gone for two years in a row to achievements in astrophysics and astronomy.  Following the same pattern, each year one theoretical astrophysicist and two observational astronomers shared the prize.  In 2019, the theorist James Peebles essentially received a lifetime achievement award for his contributions to the standard model of cosmology, while Michel Mayor and Didier Queloz were honored for the first discovery of an exoplanet orbiting a solar-like star.  This year, Roger Penrose was honored for his theoretical work on black holes.  Laureates Reinhard Genzel and Andrea Ghez are the heads of two competing groups making observations at Sagittarius A*, who determined that the center of the Milky Way galazy is occupied by a "supermassive compact object".  This last is carefully worded; the obvious implication is that this object is a Black Hole, but the Nobel Committee chose not to use those words.  I am pleased that the leads of these fiercely competitive groups were honored together.

Finally, it is notable that though the third year of the Nobel Prize in Physics included its first female laureate (Marie Curie, 1903), a 60 year gap occurred before the second (Maria Goeppert Mayer, 1963), and a 55-year gap occured before the third (Donna Strickland, 2018).  Only a two-year gap separated the third and fourth women laureates.  I started the study of physics 28 years ago and saw year after year pass with no female laureates, while several possible female candidates passed away, like Deborah Jin who died at age 47.  It's therefore breathtaking to see two female laureates in just the last two years!  Hopefully this is not simply a 'market correction' but rather a 'new normal' (apologies for the cliche'd analogy).

 

Chemistry

 

First I can't resist bragging that the 2014 prize was awarded to physicists.  Most of the recent prizes have been for technological innovations like theirs, with many being specific to the life sciences.  This is befitting chemistry's status as the "central science", overlapping on one end with physics and at the other with biology and medicine.  Finally, it is notable that 4 of the 7 ever female laureates were awarded in the last 11 years (Ada Yonath, 2009; Frances Arnold, 2018; and this year's laureates Emannuelle Charpentier and Jennifer Doudna).

Before this year, each woman laureate in physics or chemistry had died before the next one was awarded in her field; most tragically, Irene Joliot-Curie winning the year after her mother died.  The exception was Frances Arnold's 2018 award while Ada Yonath was still alive.  However now there are four living female laureates in chemistry and two in physics, an unprecedented occurrence, but let us hope these numbers will only increase moving forward.  

 

(By my count, there are currently 7 living female laureates in physiology and medicine; 10 in peace; 6 in literature; and only 1 in economics, though that prize has a shorter history than the others.)

Matt Ridley on "What the Pandemic has taught us about Science"

This weekend's issue of the Wall Street Journal features an insightful essay by Matt Ridley, in the Review section, titled "What the Pandemic has taught us about Science".  Two of the best passages are the following very simple observations:

"Seeing science as a game of guess-and-test clarifies what has been happening these past months.  Science is not about pronouncing with certainty on the known facts of the world; it is about exploring the unknown by testing guesses, some of which prove wrong."

"The health of science depends on tolerating, even encouraging, at least some disagreement.  In practice, science is prevented from turning into religion not by asking scientists to challenge their own theories but by getting them to challenge each other, sometimes with gusto."

I strongly recommend the article to DTLR readers, even if you may nit-pick a few details.

The solar corona

We've been hearing a lot about "corona" lately, but a remarkable photo of the solar corona has been making the rounds, courtesy of the European Space Agency's Solar Orbiter satellite mission.  (The link is to Elizabeth Gibney's Nature article about it.)  The photo is an excellent complement to a high-resolution photo of solar granulation released earlier this year by the ground-based Daniel K. Inouye Solar Telescope in Hawaii (see Alexandra Witze's Nature article here.)  Along with NASA's Parker Solar Probe, and a new European ground-based telescope planned for the Canary Islands, we may be entering an exciting period to do solar physics.

Escape From Model-Land

Earlier this week, Mark Buchanan's column in Nature Physics featured "The Limits of a Model" , concerning  the pitfalls of epidemiological forecasting, of course a topic very much in the news in the last few months.  Near the end Buchanan cites an intriguing paper by Thompson & Smith (2019), "Escape From Model-Land", a bracing discussion of why one should be very humble about using mathematical/empirical models.  It calls to mind another entertaining piece from over a decade ago, the "Financial Modeler's Manifesto" by Emanuel Derman and Paul Wilmott.

DTLR strongly recommends the Thompson & Smith piece, a candidate for required reading by every applied mathematician, applied statistician, and mathematical/computational modeler in every discipline.

Reference

Erica L. Thompson and Leonard A. Smith, 2019:  Escape From Model-Land.  Economics:  The Open-Access, Open-Assessment E-Journal, 13:  2019-40.

A critique of "AI Feynman"

Udrescu & Tegmark (2020) describe a neural network-based machine learning algorithm for "symbolic regression", which they define as determining the form of a functional relationship between a response variable and a list of input variables from a data set.  Kepler's realization that Mars' orbit is elliptical, inferred from observational astronomy data, is the prime example they give.  Their "AI Feynman" algorithm has a number of modules, including automated dimensional analysis, polynomial fitting, neural network-based data interpolation, tests for symmetry and separability, and so on.  The paper concludes with the ambitious declaration, "We look forward to the day when, for the first time in the history of physics, a computer, just like Kepler, discovers a useful and hitherto unknown physics formula through symbolic regression!"  While the paper reports important accomplishments and leaves the door open to further improvements in the authors' methodology, DTLR believes that the effort is less than meets the eye.  Here I will discuss some methodological weaknesses, most of which the authors acknowledge, as well as a conceptual weakness.

Methodological weaknesses

The authors compare their "AI Feynman" algorithm to a commercial software called Eureqa.  They defined a set of 100 algebraic equations from the Feynman Lectures and compared the two algorithms' ability to discover the correct functional relationship, using simulated data from the "true" relationship (without and with Gaussian noise in the response variable, but not in any of the independent variables).  Data for the input variables was sampled uniformly between 1 and 5, a rather arbitrary choice.  The authors triumphantly report that AI Feynman had a 100% success rate, while Eureqa achieved only 71%.  However, buried in the paper is the fact that they used the Feynman Lectures as a training set, on which their algorithm was tuned (ie, potentially overfit to).  Thus the comparison with Eureqa on this data set is wholly unfair, and the claimed 100% success rate is not generalizable.  Recognizing this, they define a set of 20 equations drawn from other classic physics texts, reporting that AI Feynman solved 90% while Eureqa solved only 15%.  Finally a third test set of somewhat arbitrary mathematical relationships was defined; AI Feynman achieved 67% success compared to Eureqa's 49%.  The authors explain that this last set of equations may not have the nice properties of physical laws that AI Feynman was designed to exploit.

Data from real experiments would rarely resemble the simulated data that the authors used.  At the early stages of an investigation, part of the domain of the input variables may even be experimentally/observationally inaccessible.  Thus the authors' comparison exercise seems artificial, though a necessary preliminary step before tackling real data.  Nonetheless it is important to point this out to readers who might find the reported performance metrics impressive.  The authors acknowledge that they did not include differential or integral equations in their evaluation, nor noise in the input variables, nor real data sets, though they hope to address all these weaknesses in future work.

A word about dimensional analysis:  In the authors' Newtonian gravity example (their Fig. 2), why were the temperatures of the masses not considered as candidate input variables?  Why is gravity completely ignored in a dimensional analysis of the ideal gas law (Lemons, 2017)?  In both cases, physical intuition is the answer.  The first step in any dimensional analysis is of course the selection of candidate variables to include in the functional relationship.  In the authors' exercise, the candidate variables were pre-selected from knowledge of the true relationship.  In reality, physical intuition (which requires training; it is not the same as natural intuition) must be used.  Santiago (2019) states that this step is the most difficult and requires the most experience.  The physicist generating the experimental data has often already made the decision on which variables should be measured, so the point may be moot, at least for physics problems.  However, if the method is extended to other, more empirical domains, such as public health, the social and behavioral sciences, and finance, it will run into trouble.  It is often far less obvious which variables should be considered even as candidates for inclusion, and dimensional analysis may cease to be an operative method.  In fact, it is often the case that you have little or no data on the input variables you really need, while you have plenty of data on variables of limited relevance to your problem. 

Conceptual weakness

When Max Planck first obtained his blackbody radiation law, as an empirical formula that satisfied constraints imposed both by the known data and physical intuition, he did not publish it.  Why?  Because at that moment, the formula was strictly empirical; it did not provide any physical insight.  Planck published it only after he developed a theoretical model of resonators in thermal equilibrium with the radiation, from which the formula could be derived.  A feature of his model was the quantization of energy.  The latter was not taken seriously until Einstein showed that energy quantization could also resolve the mystery of the photoelectric effect, superfically a completely different physics problem.  The new physics wasn't just one new equation, it was a new concept.  If energy quantization were an intrinsic property of nature, it would manifest in many physics problems, and physicists gradually discovered that this was indeed the case.  Symbolic regression is at best a contributing factor to discovering new physics; it cannot be the sole tool for doing so.

References

D. S. Lemons, 2017:  A Student's Guide to Dimensional Analysis.  Cambridge University Press, Sec. 1.8.

J. G. Santiago, 2019:  A First Course in Dimensional Analysis.  MIT Press, Sec. 5.2.

S.-M. Udrescu and M. Tegmark, 2020:  AI Feynman:  a physics-inspired method for symbolic regression.  Science Advances, 6:  eaay2631.