Three Sigma results in particle physics and astronomy

Also in the same issue of Science there was a news article by Adrian Cho on experimental attempts to measure the muon's magnetic moment.  Evidently there was a hint of physics beyond the standard model in earlier measurements, reaching a statistical discrepancy of 3.5 sigma, not quite at the conventional 5 sigma threshold for discovery.  At the end of the article, theoretical physicist JoAnne Hewitt of SLAC is quoted as saying "In my physics lifetime, every 3-sigma deviation from the standard model has gone away.  If it weren't for that baggage, I'd be cautiously optimistic."

This brought to mind a piece nearly 20 years ago, also in Science, by Charles Seife, "CERN's gamble shows perils, rewards of playing the odds."  It discusses several examples of highly statistically significant findings that "crumbled into dust" due to the discovery of systematic errors.  The late John Bahcall is quoted as saying "Half of all three-sigma results are wrong.  Look at the history."  He goes on to say "It's the systematic errors that kill you."

Seife adds,

Discoveries that seem statistically unassailable can vanish overnight, while flimsier looking findings have entered the award rosters and the textbooks without cavil.  Qualitative factors, such as the reputation of a team of scientists, whether a finding conforms to prevailing theory, and how and why the team announces a discovery, can determine whether it wins the Nobel Prize or languishes as an also-ran.

Seife's piece is a good one, worth reading and saving.