Scientific software reproducibility

Writing in Nature, Erika Check Hayden describes steps being taken by the journal Nature Biotechnology to improve the reproducibility of computational results.  She cites cases where error-prone software led to the publication of incorrect results in good journals.  Lack of software documentation seems to be one of the major issues; software testing and reproducibility were also mentioned.  The article also delves into the challenges faced by such a policy, such as finding qualified reviewers, and the danger of public shaming. 

DTLR feels that such challenges should not be allowed to block efforts to reform publication standards.  I endorse the stance taken by Nature Biotechnology, which can be found here. 

Reproducible research in the Chronicle of Higher Education

Writing for the Chronicle of Higher Education last month, Paul Voosen covers the National Institutes of Health's work in encouraging reproducible research.  (The article is behind a pay-wall, so I have not linked it here.)   The NIH seems to be getting its act together.  The article points to universities, however, as the weak link.  "Indeed, more than any part of the scientific system, the universities have been ignoring the replication crisis," the article states, attributing the thought to Glenn Begley, author of the well known Amgen study of non reproducible research (Begley & Ellis, 2012).  This has the ring of truth to me.  As I've stated before, I commend the NIH for finally acknowledging the issue and taking steps to remedy it, some of which are detailed in the brief article.  The NIH and other funding agencies, as well as prominent journals, must necessarily take a top-down approach.  However this needs to be complemented by a bottom-up approach, the embracing of a cultural change by scientists in the trenches.  Such a change will be opposed by those who benefit under the status quo, as Arturo Casadevall suggests in the article.  The tight and thus highly competitive funding environment exasperates the problem, as his colleague Ferric Fang notes in the article. I would argue that scientists need to remember why they became scientists, and they need to be angry about the non-reproducible research that pollutes the academic literature.  However, this is a tough thing to say when labs have to fight for funding survival.  Anger and idealism don't do much good if your lab has gone out of business and you failed to achieve tenure.  It's a conundrum.

I highly recommend Voosen's article to DTLR readers.

Reference

C. G. Begley and L. M. Ellis, 2012:   Raise standards for preclinical cancer research.  Nature, 483:  531-533.

Paul Voosen, 2015:  Amid a sea of false findings, the NIH tries reform.  Chronicle of Higher Education, March 20, 2015, page A12.

Limiting the number of mistakes when using big data

A few weeks ago, the Economist's Technology Quarterly had an excellent profile of biostatistician Dr. Susan Ellenberg in its 'Brain Scan' column.  The article describes her long and influential career in clinical trials, spanning the NIH, the FDA, and academia.   Example include her insistence that patients who did not follow the trial protocol be tracked, her championing of surrogate endpoints in cancer and HIV trials, her involving of patient groups in planning clinical trials, and her work on interim analysis and vaccine safety.

The article brings us to the present, and the benefits and hazards of using big data, which is typically observational data found in massive health care records databases, such as those owned by healthcare and insurance organizations.  Ellenberg summarizes the issues as follows:  "The more people you have the richer your database will be but also the more ways there are to be misled by the data."  The article concludes, "We've got all this data...The answer isn't to ignore it.  The answer is to figure out how to limit the number of mistakes we make."

The article does not give examples of such mistakes, but readers steeped in statistical thinking can come up with examples of their own.  Multiplicity is attached to many such mistakes; a phenomenon that can lead to identifying spurious correlations.  For instance, a blind search for correlated variables in such databases, perhaps assisted by subsetting and subgrouping, is bound to find many spurious correlations by chance alone.  Few of these would be reproduced in other data sets or in future data.  For those that are reproduced, the direction of causality, if there is one, may be unclear; alternately a lurking variable (one not measured or captured in the database) may hold the causal insights.  One way to protect ourselves from such mistakes is to use findings from big data only as hypotheses, to be confirmed by prospective, randomized, and blinded trials.  The short article goes into neither the nature of the mistakes or possible remedies.  Nonetheless, the article plays a valuable role in tamping down expectations of big data, a term that has received a great deal of hype in recent years.

Physicists as Secretaries of Defense and Energy?

This post is the result of some superficial searches on Wikipedia, prompted by the Senate confirmation of President Obama's nominee for Secretary of Defense. These musings are perhaps at the periphery of the blog's range of topics, and I will avoid commenting on the nomination's political or national security implications. Instead I will broaden the discussion to include past national security appointments as well as the Secretary of Energy post.

National security

In December, the President nominated Dr. Ashton Carter as his fourth Secretary of Defense. The Senate voted to confirm him earlier today. Carter has a Ph.D. in physics from Oxford University, and will be the second Ph.D.-level physicist to hold that office. Dr. Harold Brown, President Jimmy Carter's Secretary of Defense, was the first. In considering both physicists, one might also add Dr. William Perry, President Clinton's second Secretary of Defense, who has a Ph.D. in mathematics. All three were nominated by Democratic Presidents, and could be thought of as technocrats with extensive backgrounds in national security (including earlier stints in the DOD administration) in addition to their scientific credentials. Brown had previously served as Secretary of the Air Force, and was president of Caltech at the time of his nomination; Perry and Carter had earlier been deputy secretaries of defense. They contrast with most other recent SecDefs, who come from the political or business realms (with the notable exception of Dr. Robert M. Gates, about whom more later). Both Perry and Carter were nominated after other prominent candidates removed themselves from consideration. Perry was nominated after Vice Admiral Bobby Ray Inman famously withdrew his nomination, after initially accepting. More recently, Carter was nominated after others (Senator Jack Reed and former Undersecretary of Defense for Policy Michele Flournoy) allegedly declined to be considered. Of the three (Brown, Perry, and Carter), only Perry seems to have had actual military service. Brown has recently published his memoirs, Star Spangled Security. An account of Brown's DOD can also be found in chapter 10 of General Colin Powell's memoirs, My American Journey. 

Some other notable appointments in recent history could be mentioned. President Jimmy Carter's Secretary of the Air Force was a Ph.D. physicist, Dr. Hans Mark. President Bill Clinton's Secretary of the Air Force was prominent MIT aerodynamicist, Dr. Sheila Widnall. Both Mark and Widnall eventually returned to academia. President George W. Bush's Secretaries of the Navy were Gordon England, who had majored in electrical engineering in college, and Dr. Donald C. Winter, a physicist. Both have had careers in industry and government. The same president's second Secretary of the Army was Dr. Francis J. Harvey, a metallurgist, whose previous career had been in industry. Harvey was fired by Secretary Gates in the wake of the Walter Reed Army Medical Center scandal.

Among the presidential national security advisers, Admiral John Poindexter from the Reagan administration (and the Iran-Contra scandal) comes to mind. He has a Ph.D. in physics, having studied under Nobel laureate Rudolf Mossbauer. Among the CIA directors, one thinks of Dr. John M. Deutsch, an MIT physical chemist, former Deputy Secretary of Defense, and former Undersecretary of Energy. His career was tainted by an investigation into mishandling classified information, for which he was ultimately pardoned by President Clinton.

Energy

Both of President Obama's Secretaries of Energy are physicists. His first, Dr. Steven Chu, is a Nobel Laureate, while his second, MIT's Dr. Ernest Moniz, had previously served as an Undersecretary of Energy during President Clinton's second term. Despite DOE's obvious connection with physics, they seem to be the first two Ph.D.-level physicists to hold that office. President George W. Bush's second Secretary of Energy, Dr. Samuel W. Bodman, has a Ph.D. in chemical engineering. His career started in academia, then moved into finance, and finally government service, serving as Deputy Secretary at both Treasury and Commerce before taking the helm at DOE. Otherwise, like at Defense, most Energy Secretaries have a background in politics or business; a number have had extensive experience at DOD as well. From this perspective, Chu and Moniz seem again to be technocrats like Brown, Perry, and Carter at Defense.

Other physicists

Like the vast majority of presidential science advisers, Obama's is a physicist, Dr. John Holdren. For a while, a Nobel laureate physicist served in a senior position in Holdren's office, Dr. Carl Wieman. The current director of the National Science Foundation (NSF), Dr. France Cordova, is an astrophysicist; she is the former president of Purdue University. The lead positions of agencies such as the Office of Science and Technology Policy, the NSF, the NIH, and the CDC, are almost always held by scientists.

What to make of all this?

President Obama's cabinet will be unique in having two Ph.D.-level physicists serving simultaneously on it. (Both Carter and Moniz are also fellows of the American Physical Society.) I am not aware of physicists serving in any other cabinet-level office in recent history besides DOD and DOE. (President Clinton's attorney general, Janet Reno, was a chemistry major in college.) The Obama cabinet also has, as its second Interior Secretary, Sally Jewell, who has a bachelor's degree in mechanical engineering. She started her career in the oil industry, and later moved into banking.

Cabinet appointments are usually political, and technocrats (like the Ph.D.-physicists under discussion) seem to be in the minority. Looking at other cabinet offices within the purview of this blog, we see that at Health and Human Services, evidently only one medical professional has actually ever served as Secretary since the department was separated from Education: Dr. Louis Sullivan, under the first President Bush.

The preceding comments have been largely factual; what follows is opinion and speculation. It does not seem to me that a background in science or technology would necessarily add or subtract from the qualifications of a cabinet level official. Given the role of technology in the armed forces and in energy, it does not surprise me that, among the many career paths that lead to a cabinet level position in defense or energy, science or engineering might be included. However, such a background is neither necessary nor sufficient. Arguably the finest SecDef in recent memory, Dr. Robert M. Gates, did not have a background in science or engineering. His doctorate was in Russian and Soviet history.

The appointment of Gates is nearly as exceptional as the appointment of Ph.D.-technocrats, for like them Gates represents a small number of SecDefs who have had primary careers in national security, as opposed to political and business leaders (with perhaps particular expertise in national security).

Disclaimer: I have not considered Acting Secretaries in the above account. Moreover, the use of Wikipedia has naturally limited the accuracy of the information I report above. Readers are invited to submit corrections or other perspectives in the Comments.

References

Harold Brown with Joyce Winslow, 2012: Star Spangled Security. Brookings Institution Press.

Colin Powell with Joseph E. Persico, 1995: My American Journey. Random House (New York).

More on Reproducible research

Earlier this week, Joel Achenbach covered reproducible research in a Washington Post article.  This follows hot on the heels of the Science News series mentioned in my last post.  This is a topic that has received much discussion within the scientific literature, such as in Nature and Science, and bled into the popular press, for instance, with an Economist cover story in the fall of 2013, and a National Public Radio piece last fall by Richard Harris.  Achenbach's article doesn't have anything particularly new for those who have been following this thread over the last few years (including readers of this blog).  However, this topic deserves attention from major news organizations such as the Post and the Economist.  The taxpayers, after all, are bankrolling much of scientific research, and deserve to be kept in the loop on how their money is spent, or mis-spent as the case may be.

I also want to call readers' attention to an opinion piece last fall by John Ioannidis (2014).  It is a forward looking piece on how to make research more reproducible.  Much of the paper is focused on the infrastructure of the scientific community, including the incentive systems.  Ultimately this is indeed where change must occur.  He also has a list of "Some research practices that may help increase the proportion of true research findings."  Some of these are not explained in detail in this paper.  Third to last on his list is "Improvement of study design standards," an issue I feel is paramount.  Unfortunately Ioannidis does not go into great detail on this particular point, though it could deserve a paper all of its own.

My feeling is that, despite all the attention, reproducible research is not yet a big deal in the scientific community.  Scientists, and those who fund them, aren't angry enough yet to push for serious changes.  Until that happens, DTLR will not rest in promoting reproducible research practices.

Reference

John P. A. Ioannidis, 2014:  How to make more published research true.  PLoS Medicine, 11 (10):  e1001747.

Science news on reproducible research

Science News offers an excellent two-part examination of the woes of reproducible research by Tina Hesman Saey.  Here is part one and here is part two.  Highly recommended.

Let there be a year of light

Happy New Year, readers!  DTLR joins in celebrating 2015 as UNESCO's "International Year of Light and Light-based Technologies" (IYL 2015).  The year 2015 will also be the anniversary of several optics achievements, such as the publication of Maxwell's equations in 1865, celebrated in January's issue of Nature Photonics.  The IYL 2015 organizers also point to the following anniversaries:

• 1015:  Ibn Al Haythem's Book of Optics.
• 1815:  Fresnel and the wave nature of light.
• 1915:  Einstein's general relativity - light in space and time.
• 1965:  Cosmic microwave background; Charles Kao and optical fiber technology.

DTLR wishes readers a happy new year and Happy International Year of Light.