John R. Rice's "Future Scientific Software Systems"

Today I stumbled upon the slides of a presentation, "Future Scientific Software Systems", presented at the first IEEE Computer Society Workshop on Computational Science and Engineering, in October 1996 at Purdue University.  The presenter was John R. Rice (1934-2024), a distinguished professor of computer science at Purdue.

Some remarkable statements in the talk did not make it into the published version, which appeared the following year in the journal, IEEE Computational Science and Engineering. Slide 20, for example, states that Numerical Recipes is "Extremely successful commercially and a failure scientifically.  The lessons learned in the 1950s-1970s are ignored."  Under the bullet for Microsoft Excel, it simply says "Ditto".  Late in the slide, it says "Poor, unreliable, inefficient software can be a commercial success.  Most users have no reliable way to assess software quality."

Much has been written about the flaws of Microsoft Excel elsewhere, so I suspect there won't be much controversy over this statement.  Rice's claim about Numerical Recipes was a shock to my system when I first heard it (from Rice himself, though at a different forum). The first edition of NR (dating to 1986) was the bread and butter of the computational component of my undergraduate research project in 1994-1995; admittedly I had not had any formal training in computer science beyond high school at the time I was working on that project.  However, my elders seemed to treat NR as an authoritative source, and I had no reason to think otherwise until I heard Prof. Rice denounce it just a year or two later.  The fact that NR was written by users (albeit very sophisticated ones) rather than "professional" computer scientists or computational mathematicians, lent some plausibility to the claim.

About a decade later, when I was a working professional, I discovered empirically an example of one of NR's flaws.  I was trying to understand spectral analysis of unevenly spaced data, and NR promotes the Lomb-Scargle method.  I quickly discovered that this was a very active field of research, and there was a plethora of alternatives to Lomb-Scargle, though I didn't have the time to investigate them all and figure out which ones were the most fit for purpose for our work.  No hint of this rich literature is made in NR, not even in the third (2007) edition.  I had finally discovered evidence that what Rice had said a decade earlier might not just be the opinion of a disgruntled academic.  On the other hand, the community of "professionals" could be considered as having equally failed to provide a pragmatic alternative to NR to users.  

Returning to Rice's presentation, the slides after slide 20 start to look ahead in time.  The final slide (slide 33) is amusing as it is titled "High impact applications that won't happen". He predicts that "By 2015 we will not have simulations that are reliable and accurate for"

• Weather forecasts of several days
• Social interactions such as the economy, small groups, warfare, business growth
• Life forms of a single cell
• Geophysics such as earthquakes and volcano eruptions
• Climate
• Software engineering of large Fortran/C applications 

Here we are ten years after 2015, in 2025.  I think the list of applications that did not happen remains accurate, except possibly for the first bullet (weather forecasts).  He does not define how "reliable and accurate" such forecasts need to be, but I think it would be unfair to claim that we cannot make such forecasts reasonably well in 2025.

Rice looked ahead 20 years.  Could we see progress on the rest of the above list in the next twenty?  One thing Rice did not foresee is that simulations might come to be supplanted by artificial intelligence methods, such as large language models. We have already started to see AI encroach as a competitor to simulations in weather forecasting.  Alternatively, simulations and AI might find a way to work together to make progress on these kinds of problems.  I'm reluctant to be completely skeptical that no progress on the above list will be made in the next 20 years, though not necessarily by simulations alone.

 

Reference

J. R. Rice, 1997:  Future scientific software systems.  IEEE Computational Science and Engineering, April/June issue, pp. 44-48. 

The 2024 Nobel Prizes in science

DTLR cannot resist opining on this year's Nobel science prizes laureates, especially when several of the non-physics laureates have physics connections, while one of the physics laureates does not.

One of the two laureates for physiology or medicine is Gary Ruvkun, whose bachelor's and Ph.D. degrees are in biophysics.  One of the three laureates for chemistry, David Baker, had done a postdoc in biophysics.  He currently has an adjunct appointment in the physics department at his university, along with four other departments aside from his home department.  A second chemistry laureate, John Jumper, had done a bachelor's degree in physics and math, and a master's in theoretical condensed matter physics, before earning a second master's and a Ph.D. both in theoretical chemistry.

It is perhaps also notable that David Baker's father was a physicist and his mother was a geophysicist.  One of the physics laureates, John Hopfield, also had two physicist parents.  

Artificial intelligence was the center of this year's physics and chemistry awards.  Two of the three chemistry laureates were awarded for AI-based protein structure prediction, for their work with AlphaFold, which is owned by Google's parent company, Alphabet.  There is little doubt that this achievement belongs to biological chemistry.  I was pleased to see a share of the award go to employees of a private sector company instead of a university, not something that happens often nowadays.

The physics prize, however, has been extremely controversial, in many cases sparking outrage and cynicism, among both physicists and non-physicists.

Let's start with the laureates themselves before discussing their achievements.  A number of commentators, including some who should know better (e.g., Sabine Hossenfelder) falsely claim that the award was not given to physicists, but to computer scientists.  This is only half true.  One of the laureates, Geoffrey Hinton, is not a physicist, by either education or employment.  He is a cognitive scientist and computer scientist.  (Apparently he is also a direct descendent of George Boole!)  The other, John Hopfield, has superb physics bona fides.  His education was in physics, he was a Buckley Prize winner for his contributions to condensed matter physics, and he is probably the greatest living biological physicist today.  It is true that he is also considered a neuroscientist and a molecular biologist.  In fact, his last academic position before retirement was on the Princeton University molecular biology faculty.  He is a titan of both physics and interdisciplinary science, though the Nobel committee focused on his work on Hopfield Networks, a model of associative memory based on statistical physics (specifically, spin glasses).  He has served as President of the APS.

Unlike many, I celebrate Hopfield's Nobel award.  Geoffrey Hinton is really the puzzle here.  The Nobel committee zeroed in on his stochastic versions of Hopfield Networks, which are called Boltzmann machines.  However, his contributions to neural networks go well beyond that:  he introduced the backpropagation algorithm and later was one of the founders of deep learning methodology.  He is just as much a titan in the neural net community as Hopfield is in the biological physics community.

The history of neural networks is long, and boasts key contributions by a number of individuals.  While Hinton in undoubtedly a key figure in this history, others have also been recognized (including by the Turing award).  Perhaps the key point for the Nobel committee was Hinton's use of statistical physics ideas for the Boltzmann machine. 

There is no doubt that neural networks are used in physics, as in so many other fields of science and engineering.  There is also no doubt that ideas from physics have influenced neural network technology, and machine learning more generally.  But are Hopfield Networks and Boltzmann machines worthy of a Nobel Prize in physics?  This is a question that was hotly debated in the days after the awards were announced.  It seems though, the debate has been lopsided, at least in the sources I've read, with few commentators being willing to defend the Nobel commitee's choice this year.  This award was a huge surprise to nearly everyone who follows the Nobel Prize news every October, including me.  I was even more shocked when the host of the Physics World podcast admitted he had never heard of John Hopfield until the announcement of the prize.

The American Physical Society, in its official new release, decided to get behind the award, but notably emphasized Hopfield as a longtime member of the APS community.  This could be read as a veiled critique of Hinton's award; as far as I know, Hinton's one contribution to the physics literature is a co-authored paper that was published in a physics conference proceedings.

In principle there are several APS units that might choose to celebrate the award to either Hopfield or both Hopfield and Hinton:  the Division of Biological Physics, the Division on Computational Physics, the Topical Group on Statistical and Nonlinear Physics, the Topical Group on Data Science, and the Forum on Industrial and Applied Physics.  However, I have seen and heard of few who have actually embraced this year's award.

This is regrettable.  I think Hopfield is deserving of the honor in the sense that James Peebles was in 2019 - a kind of lifetime career recognition.  I'm not ready to render an opinion on Geoffrey Hinton's Nobel Prize in Physics.  We need to take a step back from the knee-jerk, immediate, and highly emotional reactions.  I'm hoping some thoughtful commentators will, in the coming year, weigh in with well considered opinions.  As a physicist who has worked in machine learning (there are now many, many of us), I may choose to view this award as recognition of those working in this interdisciplinary space who actually use physics ideas, regardless of our formal education or employment in physics.

Springer's Lecture Notes in Computer Science celebrates 50th anniversary

The Verlag von Julius Springer, now called Springer Nature, is celebrating the 50th anniversary of the series Lecture Notes in Computer Science this year.  As I write this, there are over 13,800 volumes published in the series, though most of the earliest ones are long out of print.  The series also boasts two subseries, Lecture Notes in Artificial Intelligence, and Lecture Notes in Bioinformatics.  Most of the volumes in LNCS are actually meeting/conference proceedings.  Indeed, almost 20 years ago I co-authored a paper published in the refereed proceedings of a workshop that appeared in LNCS.  In the nearly 2 decades since, that paper has garnered well over 300 citations, according to Google Scholar.  Based on that n=1 case, I would say that the series successfully disseminated the research we reported in our contribution.

DTLR congratulates Springer on the 50th anniversary of LNCS.

Escape From Model-Land

Earlier this week, Mark Buchanan's column in Nature Physics featured "The Limits of a Model" , concerning  the pitfalls of epidemiological forecasting, of course a topic very much in the news in the last few months.  Near the end Buchanan cites an intriguing paper by Thompson & Smith (2019), "Escape From Model-Land", a bracing discussion of why one should be very humble about using mathematical/empirical models.  It calls to mind another entertaining piece from over a decade ago, the "Financial Modeler's Manifesto" by Emanuel Derman and Paul Wilmott.

DTLR strongly recommends the Thompson & Smith piece, a candidate for required reading by every applied mathematician, applied statistician, and mathematical/computational modeler in every discipline.

Reference

Erica L. Thompson and Leonard A. Smith, 2019:  Escape From Model-Land.  Economics:  The Open-Access, Open-Assessment E-Journal, 13:  2019-40.

A physicist attends JMM

The Joint Mathematics Meetings (JMM) are advertised as the world's largest mathematics conference. This year's meeting was held in San Diego, CA, last week. DTLR attended despite his mild allergy to pure mathematics. This was my first time attending JMM, and I chose the sessions I attended with great care, and learned a lot. Some highlights are discussed below.

San Diego Convention Center, site of JMM 2018.

Physics

The best talk given by a mathematician was a physics talk, “Toy models,” by Stanford professor Tadashi Tokieda. He used a series of toys exhibiting unexpected, puzzling, and surprising behavior to illustrate ideas in physics, using almost no mathematics at all. Rather, he relied on qualitative reasoning and dimensional analysis rather than direct computation. After I returned home I discovered that many of his examples, and others, can be found in his Youtube videos, some of which are collected here. He was an exceedingly entertaining speaker – I was not bored for even one second. I would rate this as the best talk of the conference.

Computer scientist Dana Randall (Georgia Tech) gave a presentation on statistical physics, “Emergent phenomena in random structures and algorithms.” She discussed phase transitions in lattice gases, randomized algorithms, and swarm robotics, among other topics. (Phase transitions were also one of the topics addressed by Tokieda.)

Fluid Dynamics

Edriss S. Titi (Texas A&M and Weizmann Institute) provided a review of mathematical results on existence, uniqueness, and regularity of solutions for the incompressible Euler and Navier-Stokes equations under various conditions, such as 2D vs. 3D flows, and types of initial conditions. This is of course the subject of one of the million dollar prizes offered by the Clay Mathematics Institute. Rayleigh-Benard convection was given as an example. 

Isabelle Gallagher (University of Paris Diderot) presented a review of mathematical results connecting the Newton hard-sphere gas model to the Boltzmann transport equation from the kinetic theory of gases, and to the Navier-Stokes equations for a continuum fluid. One of the puzzles is how does a fundamentally reversible system – Newton's laws applied to a gas of hard spheres – result in irreversible behavior characterized by the second law of thermodynamics (reflected in both the Boltzmann and Navier-Stokes models). Her answer to this is the Ehrenfest experiment, where such a gas begins in one chamber, and at a certain time the portal to a second chamber is opened. Eventually an equilibrium is reached where both chambers have approximately the same number of particles. The key is the number of particles. If there are just two particles, nothing remarkable is observed. However, when there are many particles, the most likely states of the system are those near the equilibrium state. Thus, the statistical properties of a completely deterministic system are consistent with the second law of thermodynamics. We might think of this as an emergent behavior, not unlike those discussed in Dana Randall's talk.

Data Analysis

Topologist Gunnar Carlson (Stanford) discussed “Topological Modeling of Complex Data”. Here the “model” is not a statistical model, but rather a network model that attempts to capture the shape of data. The idea is to apply overlapping bins to the data along some of the predictor variables, and cluster the data along these axes into nodes. Nodes with overlapping data are connected, forming a network graph.

Applied mathematician Tamara G. Kolda (Sandia National Labs) spoke about tensor decompositions, particularly a decomposition known as CP (canonical polyadic). These tensor decompositions are not  orthogonal, but the idea is to essentially project high dimensional data onto what I will call basis tensors. Randomization plays a key role in the algorithm.

Neither of these presentations is by a statistician, and neither addresses statistical inference, rightly so in my view. Rather, they belong to exploratory data analysis (EDA), a field that Carlson reminds us was invented by (topologist) John Tukey.

Computer Science

Harvard computer scientist Cynthia Dwork (joint appointment with Microsoft) presented a talk on differential privacy. Algorithms that exhibit differential privacy are randomized algorithms that respond to a query to a data base, and provide the following guarantee. A specific person's decision to be included (or not) in the data base should not affect the outputs of the algorithm. Let two data sets differ by a single individual's record (she is present in one data set, and a different randomly sampled person from the population replaces her in the second). Any outcome output from the algorithm run on either data set will be almost equally likely. Algorithms with the differential privacy property are also inherently robust in the statistical sense, and can be used for adaptive/exploratory reuse of the data for statistical modeling.

A theme is evident: the talks by Randall, Kolda, and Dwork all involve randomized algorithms in computer science, not a topic that I thought about when I was a student 20 or so years ago.

What about Math?

I did attempt to attend some actual pure math talks: Alissa Crans (Loyola Marymount University) on “Quintessential quandle queries” and Craig Huneke (Virginia) on “How complicated are polynomials in many variables”. I had not heard of quandles before; the speaker related them to both groups and knots. An application to cryptography was mentioned but not dwelled on. She did at least bring a prop (a giant tetrahedron) that she used for illustration. No applications were mentioned at all in the polynomial talk, which focused on something called Stillman's conjecture. I also attended a talk by a distinguished historian of mathematics, Joseph Dauben (CUNY), on the history of Chinese math (actually, the history of Chinese historians of mathematics).

A slide from Alissa Crans' lecture on quandles, showing the definition and the mathematician who coined the term.

I also attended a panel session on careers in business, industry, and government, which probably could have run longer than it did. I was pleased to see a high level of interest in this session, but sad that many math students aren't sure how or even whether to pursue such careers. It's a good thing that they came to this session, but their home departments should be doing more to stimulate interest and provide practical resources for such career development. There was also a talk by applied mathematician Stephen Hobbs (Space and Naval Warfare Systems Center) on a simple model for deploying aircraft-carrier based resources in a humanitarian aid scenario. Finally I attended a few sessions on statistical education, including one on developing a data science program within a mathematics department.

The exhibit hall was a delight, with many book and software vendors offering their wares.  The National Security Agency had a recruiting booth which featured an original Enigma encryption device, which attendees were invited to interact with (unlike museum pieces that remain behind glass).

An original ENIGMA device at the NSA booth.

Of course, San Diego is a nice spot to go in the winter, with many extracurricular delights.

Beef empanadas at a Spanish restaurant, Cafe Sevilla, in the Gaslamp District of San Diego.

Software validation in computational biology

Last month in Nature, there was a brief article by Erika Check Hayden about an experiment in peer review of scientific software being carried out by the new Mozilla Science Lab. Nine papers published in PLoS Computational Biology, selected by its editors, would have their code subject to a peer review by software engineers. The experiment and its motivation are described in the article; I also recommend reading the user comments posted at the end. (See also the earlier pieces by Zeeya Merali and Nick Barnes, published together in Nature in 2010.)  Apparently there has been some controversy, as scientists are understandably nervous about having their work subjected to a new form of review. However, scientists are not well trained in software development concepts such as version control, validation, and verification, and the code they write may become difficult to maintain, or even worse, produce undetected errors that have worked their way into published research.

The Mozilla Science Lab was introduced this past summer, and is led by Kaitlin Thaney. It sponsors Greg Wilson's Software Carpentry; I strongly recommend having a look at the latter's website. I've read Wilson's essays in Computing in Science and Engineering and other publications over the years, and have been sympathetic to his views. I've heard rumors that the some of the code at Fermilab is spaghetti code, with bits and pieces of it written by many hands over many decades. Such an unwieldy mass of legacy code is almost impossible to maintain. I was told about one bug whose fix generated another, more serious bug that was impossible to debug. It was decided to restore the original bug and leave it in the code!

I am fortunate in that one of my formative experiences was an internship with a small company that, as a matter of survival, implemented a fairly disciplined software construction methodology, based in part on Steve McConnell's Code Complete. Because the company was small and had a certain rate of turnover, all of their software had to be highly maintainable, assuming the original coder was no longer employed at the firm. It was a point of pride there that you wouldn't be able to tell who wrote a piece of code found in the software they developed, without looking at the header (which had version control data), for we all conformed to the same software style.

DTLR endorses the Mozilla experiment in peer review of software. I hope we learn a lot from their experiment, even if it is deemed to be a failure in the end. In a letter to the editor, Alden and Read (2013) state that software quality should be built in from the beginning, before any data are taken, and not “inspected in” at the peer review stage. They are of course right, but to protect the rest of the community I do think software peer review is a concept that should at least be explored.

References

Nick Barnes, 2010: Publish your computer code: it is good enough. Nature, 467: 753.

Zeeya Merali, 2010: Computational science:...error. Nature, 467, 775-777.

Erika Check Hayden, 2013: Mozilla plan seeks to debug scientific code. Nature, 501: 472.

Kieran Alden and Mark Read, 2013: Scientific software needs quality control. Nature, 502: 448.

Welcome to “Tide” and “Gyre”, supercomputers for operational weather forecasting

Last thursday, the National Weather Service (NWS) brought online its two new “clone” IBM supercomputers, Tide and Gyre, for operational weather forecasting. The machines are the workhorses of the NWS National Centers for Environmental Prediction (NCEP)'s Weather and Climate Operational Supercomputing System (WCOSS). The two identical supercomputers are located in different places (primary machine at Reston, VA; backup at Orlando, FL).

Patrick Thibodeau's write-up at Computerworld seems to have led media coverage of the event. He refers to last year's Hurricane Sandy, which engendered “a belief that the European Center for Medium-Range Weather Forecasts (ECMWF) had a better storm track model further out. Criticism over the U.S. forecasting ability has followed post Sandy.” Bringing the new computers online, replacing the previously used pair of IBM supercomputers, is a step in the right direction. (A planned phase II of the transition will lead to further improvements.) However, the U.S. will also need to maintain investments in Earth-observing weather satellites, as noted (for instance) by Stephen Stirling of the New Jersey Star Ledger.

The NWS computers do work of national importance, and it is reassuring to read about the verification and validation of the systems prior to going live. Further details can be found in this post by Steve Tracton at the Washington Post. He describes sensitive dependence on initial conditions, manifested as a divergence in forecasts from the same forecast model run on both the old and new supercomputers. This phenomenon, as he points out, is closely related to the path-breaking work on chaos in meteorology and computational science by the late Edward N. Lorenz (1963, 1989).

Update (29 Aug 2013):  see also the NOAA press release.

 References

 

E.N. Lorenz, 1963: Deterministic nonperiodic flow. J. Atmos. Sci., 20: 130-141.

E. N. Lorenz, 1989: Computational chaos -- a prelude to computational instability.  Physica D, 35: 299-317.

Reproducible research in computational science and engineering

The Diffusion Tensor Literary Review (DTLR) endorses the principles of reproducible research outlined in the final report of the ICERM Workshop on Reproducibility in Computational and Experimental Mathematics.  A summary of the findings was published last month by Stodden, Borwein, and Bailey (2013).

The message seems to be that computational science has failed to maintain standards of reproducibility expected in theoretical and experimental science, let alone good software engineering principles.  This has led to a "credibility crisis".  The workshop participants agreed on three principles, which I quote verbatim from Stodden, et al. (2013):

1. "It is important to promote a culture change that will integrate computational reproducibility into the research process."
2. "Journals, funding agencies, and employers should support this culture change."
3. "Reproducible research practices and the use of appropriate tools should be taught as standard operating procedure in relation to computational aspects of research."

Read the final report of the workshop:  it is surprisingly direct, candid, and eloquent for a committee-produced document.  I have not studied the associated wiki in detail, but the report establishes the big picture.  We need to get the scientific/engineering community on board with these basic principles.  It is difficult to understate that a major cultural change is indeed called for, as the current infrastructure actively discourages scientists from implementing principles of reproducible research.

Personally, as a taxpayer and as a scientist, I find the current state of affairs disgraceful.  Scientists and engineers typically spend other people's money (usually at the expense of the taxpayer) to do their work.  Ensuring that results are reproducible seems to be a minimum expectation for publication, yet in computational science and engineering it is not.  Often scientists cannot even reproduce their own work, let alone the work of others.  We need to be better caretakers of the limited amount of money that society collectively has to spend on science and engineering.

Reproducible research is a theme I plan to return to in future posts on DTLR.  Watch this space for more.

 Reference

V. Stodden, J. Borwein, and D. H. Bailey, 2013:  "Setting the default to reproducible" in computational science research.  SIAM News, 46 (5):  4-6.