Surface tension and biology

This blog has tended to focus on methodological issues in science and medicine, but occasionally I do want to lavish praise for substantive work.  Here I'd like to call readers' attention to the delightful article in Science a couple weeks ago by Elizabeth Pennisi, "Water's Tough Skin."  It is a feature article describing a number of ways that surface tension is important in biology, including for plants, animals, and microbes.  A number of scientists, engineers, and mathematicians are interviewed about their work.  This is the kind of article that reminds us of why we became interested in science, engineering, and medicine in the first place.  I won't review the content here, but I commend it to readers to enjoy for themselves.

Reference

Elizabeth Pennisi, 2014:  Water's tough skin.  Surface tension is a force to be reckoned with, especially if you are small.  Science, 343:  1194-1197.

The perils of sharing data

One of the cardinal principles of reproducible research is the sharing of data.  Making data (not just a summary of results) available is just as important for disclosure as the study design, and materials and methods.  Disclosure of the actual data permits others to confirm the analysis of the original authors, or conduct alternative analyses.

Last week's issue of Nature had an editorial outlining the perils of sharing data.  Of greatest concern:  once a data set is published, it is easy for an armchair analyst to take it and run with it, possibly depriving the original investigators the opportunity to publish findings based on a very hard won data set.  Publication of a data set does not have the same status and prestige as publishing scientific findings from the data.

I join with Nature in encouraging discussion and debate within the scientific community to resolve these issues.  The interests of the innovators -- those who design studies and collect the data -- need to be considered before rushing into a mandatory data disclosure policy.  The article talks about the possibility of inviting the original authors to be co-authors on works derived from their data by others.  This is one of many possible options.  The infrastructure of science, including funding, tenure, and promotion policies, needs to change to accommodate and encourage data sharing.  I don't have any answers, and I suspect each field will have to structure a customized solution to suit its own situation.

Welcome to Ioannidis & Goodman's METRICS

Last week's edition of the Economist carried an article announcing the creation of a new laboratory at Stanford University, called the Meta-Research Innovation Center (METRICS), founded by John Ioannidis and Steven Goodman.  It aims to promote reproducible research and shame non-reproducible research.  Of greatest note is its "journal watch" concept, which will monitor the quality of published research.  Such a watchdog has been badly needed for decades, although I wonder how the operation will be funded.

Other activities mentioned by the article include influencing policymakers to avoid relying on shoddy research, battling publication bias, organizing conferences of other meta-researchers, and evaluating the effects of encouraging reproducible research. In other words, they'd like to see if there is any evidence that evidence-based science is better science!

DTLR has long been an admirer of Ioannidis' work, and I welcome the new center and look forward to seeing its work.

This winter's weather does not reveal anything about global warming

Last month in Science, a group of five prominent atmospheric scientists from around the country published a letter cautioning us not to interpret this winter's severe weather through the lens of climate change (Wallace et al., 2014). This winter has witnessed the dip in the polar vortex over much of the U.S., resulting in all time low temperatures; there have also been a series of heavy snowstorms on the east coast. Speaking of these events, the authors write:

Some have been touting such stretches of extreme cold as evidence that global warming is a hoax, while others have been citing them as evidence that global warming is causing a “global weirding” of the weather. In our view, it is neither.

As climate scientists, we share the prevailing view in our community that human-induced global warming is happening and that, without mitigating measures, the Earth will continue to warm over the next century with serious consequences. But we consider it unlikely that those consequences will include more frigid winters.

Although such a hypothesis has been proposed, the authors do not find it corroborated with either “alternate observational analyses” nor climate model simulations. Moreover they “do not view the theoretical arguments underlying it as compelling.” The authors caution about mistaking coincidence for causation, and although they believe such hypotheses “deserve a fair hearing”, the authors seem to imply that this one is too half-baked to be made “the centerpiece of the public discourse on global warming.” They conclude:

Even in a warming climate, we could experience an extraordinary run of cold winters, but harsher winters in future decades are not among the most likely nor the most serious consequences of global warming.

Reference

John M. Wallace, Isaac M. Held, David W. J. Thompson, Kevin E. Trenberth, and John E. Walsh, 2014: Global warming and winter weather. Science, 343: 729-730.

A call for more reproducible research in drug discovery/development

Phase III drug trials are typically randomized, blinded, controlled clinical trials.  However, last month in JAMA, Djulbegovic et al. (2013) argued "more than 80% of phase 1 studies and more than 50% of phase 2 studies are currently nonrandomized."  They argue that these early phase studies should all be randomized, and that even preclinical studies in animals and cell cultures should also be randomized.  (Randomization is probably even less prevalent in preclinical research than in the early phase studies discussed in the quote.)  In other words, the authors advocate study designs that encourage reproducibility across the spectrum of clinical and preclinical research.

They argue that non-randomized studies can easily lead to incorrect decisions, both pro and con.  Thus randomized studies are more efficient and provide stronger backing for decision making.  (They also make the case that randomized studies are more ethical; I'm not sure I find their reasoning here as compelling.)  They hope that use of more rigorous study designs across the drug development arena could be one way to address the industry's infamously high failure rate.

Here is a key passage from the article, describing the literature in preclinical research.

This literature yields an excess of statistically significant findings that cannot be eventually replicated, let alone translated into clinical successes.  For preclinical research conducted by the industry, routine adoption of rigorous randomized designs should be straightforward--no company wants to spend millions of dollars for the clinical testing of useless treatments.  In fact, industry researchers have taken the lead in raising the concerns about the reproducibility of preclinical research and suggesting partial solutions.  For preclinical research conducted by non industry researchers, similar rigorous practices can also be routinely adopted and requested.  Funders and journals can specify that they will sponsor and publish animal studies only if they fulfill rigorous randomization criteria.  Justified exceptions to this rule are likely to be rare.

(I have not included the footnotes; see the original.)  The major lesson for me in the above passage is that the main contribution of statistics to such studies is in the design, not the analysis.  "Statistically significant" findings by no means guarantee that the study has a chance of being reproduced, whereas good study design principles would greatly enhance the likelihood that such studies are reproducible.  Unfortunately much of the teaching and practice of statistics, both by statisticians and non-statisticians, tends to emphasize the mathematical/calculational side, rather than the study design side.

I'd like to see a devil's advocate's response to all this.  I find the authors' views compelling, and have difficulty imagining the grounds for which one might disagree.

Reference

Benjamin Djulbegovich, Iztok Hozo, and John P. A. Ioannidis, 2013:  Improving the drug development process:  more not less randomized trials.  Journal of the American Medical Association, 311 (4):  355-356.

Responses to "When Mice Mislead"

This past week's issue of Science (the Jan. 24, 2014 issue) has two letters to the editor, responding to a report last November, "When Mice Mislead" by Jennifer Couzin-Frankel, which I discussed in an earlier post.  The first letter, by Richard Traystman and Paco Herson, points to earlier findings, similar to those reported by Couzin-Frankel, in the stroke research community.  Most importantly, they assert that "It is unlikely that poor methods used in animal studies account for all the negative clincial trials that have been performed based on preclinical studies.  After all, some investigators do perform appropriate experiments, and even those studies rarely lead to positive clinical trials."  The authors point to the fact that mouse studies are usually done with healthy young mice, whereas human subjects in neuroprotective drug clinical trials are often older and have many co-morbidities.  They propose that aged mice with comorbid diseases be used in stroke trials, as a better animal model of human disease.

The second letter is from statistician Gary Churchill.  He zeroes in on one key question:  "Was the result replicated in more than one genetic background?"  He goes on to identify two "root causes" for nonreproducible research:

Science today is driven by an incentive system that often rewards precedence and impact over quality of the work.  Statistical training of scientists often emphasizes analytical techniques over experimental design and quantitative reasoning.  These are systemic problems that will not change without substantial effort

Meanwhile, Churchill endorses the message of Couzin-Frankel's article with his maxim:  "Be wise, randomize."

I think that both of these letters add value to the original piece by Couzin-Frankel. In particular, Churchill's second "root cause" is particularly interesting, as both statisticians and lay scientists or mathematicians who teach statistics are all guilty of overemphasizing methodology, modeling, and inference at the expense of study design and critical thinking. 

References

Jennifer Couzin-Frankel, 2013: When mice mislead. Science, 342: 922-925.

Richard J. Traystman and Paco S. Herson, 2014:  Misleading results:  translational challenges.  Science, 343:  369-370.

Gary Churchill, 2014:  Misleading results:  don't blame the mice.  Science, 343, 370.

The value and place of prespecifying data analysis plans

DTLR does not usually stray into the social sciences, but a paper in Science last month (Miguel, et al., 2013) provides another opportunity to dwell on reproducible research. Prospective, designed experiments are becoming more common in the social and behavioral sciences, particularly in economics and program evaluation. However, as in the natural sciences, “Commentators point to a dysfunctional reward structure in which statistically significant, novel, and theoretically tidy results are published more easily than null, replication, or perplexing results.” Reporting standards in social science journals are similarly lax as those in biology journals, and “researchers have incentives to analyze and present data to make them more 'publishable,' even at the expense of accuracy.” Examples of poor practices include the publication of positive results which form a subset of a larger study with mixed or null results, as well as presenting exploratory findings dressed up as confirmatory results.

The authors propose that three core practices be emphasized: disclosure, registration and preanalysis plans, and open data and materials. These concepts are familiar to those who work in clinical trials. However, the authors believe that the situation can be improved yet further than in the medical trial model. In the latter, the “dominant role” of government regulatory agencies “arguably slows adoption of innovative statistical methods.” The authors are also resistant to a “one-size-fits-all” approach for trial registration, preferring a method-specific approach. They foresee some convergence between methods used in behavioral research with those in medical trials, particularly in the neuroscience arena.

Near the end of the paper, there is a particularly eloquent passage that I'd like to quote in full.

The most common objection to the move toward greater research transparency pertains to preregistration. Concerned that preregistration implies a rejection of exploratory research, some worry that it will stifle creativity and serendipitous discovery. We disagree.

Scientific inquiry requires imaginative exploration. Many important findings originate as unexpected discoveries. But findings from such inductive analysis are necessarily more tentative because of the greater flexibility of methods and tests and, hence, the greater opportunity for the outcome to obtain by chance. The purpose of prespecification is not to disparage exploratory analysis but to free it from the tradition of being portrayed as formal hypothesis testing.

The above two paragraphs can easily carry over into all experimental research, not just those in the social and behavioral sciences.

Reference

 

E. Miguel, et al., 2014: Promoting transparency in social science research. Science, 343: 30-31.