Physician, Heal Thyself

Just today, Dr. Rush Holt (CEO of AAAS) sent out the following message.


DTLR does not recommend responding to this solicitation.  Until AAAS is prepared to dedicate an equal amount of resources to cleaning up Science's own house - by working to reform the perverse incentive system for scientific research, and improve its reproducibility - the above solicitation is nothing but a self-serving propaganda effort.  AAAS, as publisher of Science, the nation's leading general science journal, has a particular responsibility here, and Holt claims to be its Executive Publisher.  Will Science proceed to make the distinction between exploratory and confirmatory research, as its previous Editor envisioned?  In the midst of the crisis on reproducible research, discussed at length on this blog, how much longer will Science deserve the public's trust and support?

See also this post by Sabine Hossenfelder.  I have disagreed with her in the past, but this time I'm 100% on board.

Three Sigma results in particle physics and astronomy

Also in the same issue of Science there was a news article by Adrian Cho on experimental attempts to measure the muon's magnetic moment.  Evidently there was a hint of physics beyond the standard model in earlier measurements, reaching a statistical discrepancy of 3.5 sigma, not quite at the conventional 5 sigma threshold for discovery.  At the end of the article, theoretical physicist JoAnne Hewitt of SLAC is quoted as saying "In my physics lifetime, every 3-sigma deviation from the standard model has gone away.  If it weren't for that baggage, I'd be cautiously optimistic."

This brought to mind a piece nearly 20 years ago, also in Science, by Charles Seife, "CERN's gamble shows perils, rewards of playing the odds."  It discusses several examples of highly statistically significant findings that "crumbled into dust" due to the discovery of systematic errors.  The late John Bahcall is quoted as saying "Half of all three-sigma results are wrong.  Look at the history."  He goes on to say "It's the systematic errors that kill you."

Seife adds,

Discoveries that seem statistically unassailable can vanish overnight, while flimsier looking findings have entered the award rosters and the textbooks without cavil.  Qualitative factors, such as the reputation of a team of scientists, whether a finding conforms to prevailing theory, and how and why the team announces a discovery, can determine whether it wins the Nobel Prize or languishes as an also-ran.

Seife's piece is a good one, worth reading and saving.