Limiting the number of mistakes when using big data

A few weeks ago, the Economist's Technology Quarterly had an excellent profile of biostatistician Dr. Susan Ellenberg in its 'Brain Scan' column.  The article describes her long and influential career in clinical trials, spanning the NIH, the FDA, and academia.   Example include her insistence that patients who did not follow the trial protocol be tracked, her championing of surrogate endpoints in cancer and HIV trials, her involving of patient groups in planning clinical trials, and her work on interim analysis and vaccine safety.

The article brings us to the present, and the benefits and hazards of using big data, which is typically observational data found in massive health care records databases, such as those owned by healthcare and insurance organizations.  Ellenberg summarizes the issues as follows:  "The more people you have the richer your database will be but also the more ways there are to be misled by the data."  The article concludes, "We've got all this data...The answer isn't to ignore it.  The answer is to figure out how to limit the number of mistakes we make."

The article does not give examples of such mistakes, but readers steeped in statistical thinking can come up with examples of their own.  Multiplicity is attached to many such mistakes; a phenomenon that can lead to identifying spurious correlations.  For instance, a blind search for correlated variables in such databases, perhaps assisted by subsetting and subgrouping, is bound to find many spurious correlations by chance alone.  Few of these would be reproduced in other data sets or in future data.  For those that are reproduced, the direction of causality, if there is one, may be unclear; alternately a lurking variable (one not measured or captured in the database) may hold the causal insights.  One way to protect ourselves from such mistakes is to use findings from big data only as hypotheses, to be confirmed by prospective, randomized, and blinded trials.  The short article goes into neither the nature of the mistakes or possible remedies.  Nonetheless, the article plays a valuable role in tamping down expectations of big data, a term that has received a great deal of hype in recent years.

Physicists as Secretaries of Defense and Energy?

This post is the result of some superficial searches on Wikipedia, prompted by the Senate confirmation of President Obama's nominee for Secretary of Defense. These musings are perhaps at the periphery of the blog's range of topics, and I will avoid commenting on the nomination's political or national security implications. Instead I will broaden the discussion to include past national security appointments as well as the Secretary of Energy post.

National security

In December, the President nominated Dr. Ashton Carter as his fourth Secretary of Defense. The Senate voted to confirm him earlier today. Carter has a Ph.D. in physics from Oxford University, and will be the second Ph.D.-level physicist to hold that office. Dr. Harold Brown, President Jimmy Carter's Secretary of Defense, was the first. In considering both physicists, one might also add Dr. William Perry, President Clinton's second Secretary of Defense, who has a Ph.D. in mathematics. All three were nominated by Democratic Presidents, and could be thought of as technocrats with extensive backgrounds in national security (including earlier stints in the DOD administration) in addition to their scientific credentials. Brown had previously served as Secretary of the Air Force, and was president of Caltech at the time of his nomination; Perry and Carter had earlier been deputy secretaries of defense. They contrast with most other recent SecDefs, who come from the political or business realms (with the notable exception of Dr. Robert M. Gates, about whom more later). Both Perry and Carter were nominated after other prominent candidates removed themselves from consideration. Perry was nominated after Vice Admiral Bobby Ray Inman famously withdrew his nomination, after initially accepting. More recently, Carter was nominated after others (Senator Jack Reed and former Undersecretary of Defense for Policy Michele Flournoy) allegedly declined to be considered. Of the three (Brown, Perry, and Carter), only Perry seems to have had actual military service. Brown has recently published his memoirs, Star Spangled Security. An account of Brown's DOD can also be found in chapter 10 of General Colin Powell's memoirs, My American Journey. 

Some other notable appointments in recent history could be mentioned. President Jimmy Carter's Secretary of the Air Force was a Ph.D. physicist, Dr. Hans Mark. President Bill Clinton's Secretary of the Air Force was prominent MIT aerodynamicist, Dr. Sheila Widnall. Both Mark and Widnall eventually returned to academia. President George W. Bush's Secretaries of the Navy were Gordon England, who had majored in electrical engineering in college, and Dr. Donald C. Winter, a physicist. Both have had careers in industry and government. The same president's second Secretary of the Army was Dr. Francis J. Harvey, a metallurgist, whose previous career had been in industry. Harvey was fired by Secretary Gates in the wake of the Walter Reed Army Medical Center scandal.

Among the presidential national security advisers, Admiral John Poindexter from the Reagan administration (and the Iran-Contra scandal) comes to mind. He has a Ph.D. in physics, having studied under Nobel laureate Rudolf Mossbauer. Among the CIA directors, one thinks of Dr. John M. Deutsch, an MIT physical chemist, former Deputy Secretary of Defense, and former Undersecretary of Energy. His career was tainted by an investigation into mishandling classified information, for which he was ultimately pardoned by President Clinton.

Energy

Both of President Obama's Secretaries of Energy are physicists. His first, Dr. Steven Chu, is a Nobel Laureate, while his second, MIT's Dr. Ernest Moniz, had previously served as an Undersecretary of Energy during President Clinton's second term. Despite DOE's obvious connection with physics, they seem to be the first two Ph.D.-level physicists to hold that office. President George W. Bush's second Secretary of Energy, Dr. Samuel W. Bodman, has a Ph.D. in chemical engineering. His career started in academia, then moved into finance, and finally government service, serving as Deputy Secretary at both Treasury and Commerce before taking the helm at DOE. Otherwise, like at Defense, most Energy Secretaries have a background in politics or business; a number have had extensive experience at DOD as well. From this perspective, Chu and Moniz seem again to be technocrats like Brown, Perry, and Carter at Defense.

Other physicists

Like the vast majority of presidential science advisers, Obama's is a physicist, Dr. John Holdren. For a while, a Nobel laureate physicist served in a senior position in Holdren's office, Dr. Carl Wieman. The current director of the National Science Foundation (NSF), Dr. France Cordova, is an astrophysicist; she is the former president of Purdue University. The lead positions of agencies such as the Office of Science and Technology Policy, the NSF, the NIH, and the CDC, are almost always held by scientists.

What to make of all this?

President Obama's cabinet will be unique in having two Ph.D.-level physicists serving simultaneously on it. (Both Carter and Moniz are also fellows of the American Physical Society.) I am not aware of physicists serving in any other cabinet-level office in recent history besides DOD and DOE. (President Clinton's attorney general, Janet Reno, was a chemistry major in college.) The Obama cabinet also has, as its second Interior Secretary, Sally Jewell, who has a bachelor's degree in mechanical engineering. She started her career in the oil industry, and later moved into banking.

Cabinet appointments are usually political, and technocrats (like the Ph.D.-physicists under discussion) seem to be in the minority. Looking at other cabinet offices within the purview of this blog, we see that at Health and Human Services, evidently only one medical professional has actually ever served as Secretary since the department was separated from Education: Dr. Louis Sullivan, under the first President Bush.

The preceding comments have been largely factual; what follows is opinion and speculation. It does not seem to me that a background in science or technology would necessarily add or subtract from the qualifications of a cabinet level official. Given the role of technology in the armed forces and in energy, it does not surprise me that, among the many career paths that lead to a cabinet level position in defense or energy, science or engineering might be included. However, such a background is neither necessary nor sufficient. Arguably the finest SecDef in recent memory, Dr. Robert M. Gates, did not have a background in science or engineering. His doctorate was in Russian and Soviet history.

The appointment of Gates is nearly as exceptional as the appointment of Ph.D.-technocrats, for like them Gates represents a small number of SecDefs who have had primary careers in national security, as opposed to political and business leaders (with perhaps particular expertise in national security).

Disclaimer: I have not considered Acting Secretaries in the above account. Moreover, the use of Wikipedia has naturally limited the accuracy of the information I report above. Readers are invited to submit corrections or other perspectives in the Comments.

References

Harold Brown with Joyce Winslow, 2012: Star Spangled Security. Brookings Institution Press.

Colin Powell with Joseph E. Persico, 1995: My American Journey. Random House (New York).

More on Reproducible research

Earlier this week, Joel Achenbach covered reproducible research in a Washington Post article.  This follows hot on the heels of the Science News series mentioned in my last post.  This is a topic that has received much discussion within the scientific literature, such as in Nature and Science, and bled into the popular press, for instance, with an Economist cover story in the fall of 2013, and a National Public Radio piece last fall by Richard Harris.  Achenbach's article doesn't have anything particularly new for those who have been following this thread over the last few years (including readers of this blog).  However, this topic deserves attention from major news organizations such as the Post and the Economist.  The taxpayers, after all, are bankrolling much of scientific research, and deserve to be kept in the loop on how their money is spent, or mis-spent as the case may be.

I also want to call readers' attention to an opinion piece last fall by John Ioannidis (2014).  It is a forward looking piece on how to make research more reproducible.  Much of the paper is focused on the infrastructure of the scientific community, including the incentive systems.  Ultimately this is indeed where change must occur.  He also has a list of "Some research practices that may help increase the proportion of true research findings."  Some of these are not explained in detail in this paper.  Third to last on his list is "Improvement of study design standards," an issue I feel is paramount.  Unfortunately Ioannidis does not go into great detail on this particular point, though it could deserve a paper all of its own.

My feeling is that, despite all the attention, reproducible research is not yet a big deal in the scientific community.  Scientists, and those who fund them, aren't angry enough yet to push for serious changes.  Until that happens, DTLR will not rest in promoting reproducible research practices.

Reference

John P. A. Ioannidis, 2014:  How to make more published research true.  PLoS Medicine, 11 (10):  e1001747.

Science news on reproducible research

Science News offers an excellent two-part examination of the woes of reproducible research by Tina Hesman Saey.  Here is part one and here is part two.  Highly recommended.

Let there be a year of light

Happy New Year, readers!  DTLR joins in celebrating 2015 as UNESCO's "International Year of Light and Light-based Technologies" (IYL 2015).  The year 2015 will also be the anniversary of several optics achievements, such as the publication of Maxwell's equations in 1865, celebrated in January's issue of Nature Photonics.  The IYL 2015 organizers also point to the following anniversaries:

• 1015:  Ibn Al Haythem's Book of Optics.
• 1815:  Fresnel and the wave nature of light.
• 1915:  Einstein's general relativity - light in space and time.
• 1965:  Cosmic microwave background; Charles Kao and optical fiber technology.

DTLR wishes readers a happy new year and Happy International Year of Light.

Calculating logarithms?!

As I was culling my book collection, I came across three delightful books by Bob Miller, a CCNY math professor:  his Precalc Helper, Calc I Helper, and Calc II Helper, all published in 1991 by McGraw-Hill's Schaum division.  That was around the time I started taking calculus courses, though I do not recall using those books.  I may have acquired them shortly after I completed my year of calculus.

Regardless, as I was leafing through these books on this last day of 2014, nearly a quarter century after they were published, I was particularly struck by Miller's treatment of logarithms.  In the Precalc Helper, Chapter 12, "Modern Logarithms", begins with this paragraph (p. 75):

We will do a modern approach to logs.  Modern to a mathematician means not more than 50 years behind the times.  We will not do calculations with logs (calculations involving characteristics and mantissas).  This is no longer needed because we have calculators.  What is needed is a thorough understanding of the laws of logarithms and certain problems that can only be solved with logs.

If I ever did calculations with characteristics and mantissas, I certainly don't remember them now. It is likely that my high school had already abandoned coverage of that topic by the time I was a student.

Then on page 1 of the Calc II Helper, opening the first chapter, titled "Logarithms", we find the following passage.

Most of you, at this point in your mathematics, have not seen logs for at least a year, many a lot more.  The normal high school course emphasizes the wrong areas.  You spend most of the time doing endless calculations, none of which you need here.  By the year 2000, students will do almost no log calculations due to calculators.  In case you feel tortured, just remember that you only spent weeks on log calculations.  I spent months!!!

I suspect the future arrived a lot sooner than Miller thought it would.  When I was in high school in the late 1980s, we were using a then-new software program, Derive, to graph mathematical functions.  In college, we were using Mathematica.  When I was a teaching assistant in graduate school, graphing calculators were already pervasive, and my students were allowed to use them on exams.  (I have never owned one myself.)  Evidently graphing calculators are still in use, though equivalent apps have been available for smart phones for a few years now.  (I have never owned a smart phone either, but I suspect this will have to change one day.)

At this point I am too far out of touch with mathematics teaching and technology to know what is considered standard of practice.  Nonetheless I am grateful I never had to do endless logarithm calculations by hand.  Make no mistake, logarithms are essential for science, engineering, and medicine.  In fact, I worked with logarithms at work earlier today.  But I let the computer do the calculating.

Congratulations to arXiv

According to Nature, the arXiv preprint server has reached one million articles in its holdings.  DTLR congratulates arXiv.org, and its founder Paul Ginsparg, on this achievement.