When mice mislead

Two months ago, the Nov. 22, 2013, issue of Science announced the detection of high energy neutrinos from beyond the solar system. This of course is a major achievement for physicists. However, my attention was drawn to another story in the same issue, “When mice mislead” (Couzin-Frankel, 2013). I regard it as one of the most important works of science journalism of the year just ended.

The article documents the following “bad habits” in studies of laboratory animals, such as mice, that can lead to non-reproducible results and misleading conclusions. The bad habits discussed include:

• Removing data, such as from animals enrolled in a study but removed from the analysis for any number of reasons.
• Lack of randomization and blinding.
• No attention paid to inclusion/exclusion criteria for enrolling animals.
• Different experimental conditions for different groups of animals.
• Sample sizes too small to lead to definitive results, since researchers have very good reasons (ethical and financial) to minimize the number of animals used for research.
• Publication bias, along the lines of Ioannidis (2005).

A good example is discussed by Lisa Bero, interviewed in the article. About scientists and their mentors, Bero states that “Their idea of randomization is, you stick your hand in the cage and whichever one comes up to you, you grab. That is not a random way to select an animal.” Couzin-Frankel goes on to say that “Some animals might be fearful, or biters, or they might just be curled up in the corner, asleep. None will be chosen. And there, bias begins.”

All of these bad habits are ones that have been largely eliminated from randomized clinical trials. Lab animal studies are traditionally pursued with far less rigor than clinical studies, but the article suggests that the good habits that dominate clinical trials could really clean up preclinical research if they were to be adopted widely there too. In other words, it's time to raise the level of the game in lab animal studies, and practically it wouldn't take much additional effort to do so. In fact, this very article will help those of us who try to push back on bad habits. We now have a convenient summary of the findings that such bad habits really matter, and should be avoided.

Surprisingly, Lisa Bero found that industry funded research is less likely to endorse a drug that research funded through other means, “maybe because companies don't want to pour millions of dollars into testing a treatment in people that's unlikely to help them.” This certainly has the ring of truth; however, I think even within industrial labs, the good habits of clinical trials are not always pervasive among users of lab animals.

Joseph Bass is also interviewed with a less pessimistic view. He believes there are substantive reasons why many mouse studies fail to reproduce, such as the temperature that mice are housed at, or variation in their response with age.  However, he seems too optimistic.  Nonreproducible research is more pervasive than I would like, and while fitness for purpose as a decision criterion should always over-rule "one size fits all" rules and checklists, I think such rules and checklists will do more good than harm at this point in the history of science.

Couzin-Frankel (2013) refers to the ongoing effort by the NIH to draft rules for research it funds, to encourage openness and reproducibility, as well as the checklist for biology research promulgated by Nature last year (discussed here). She even says that Science is considering a similar policy! An NIH official is quoted as saying “Sometimes the fundamentals get pushed aside—the basics of experimental design, the basics of statistics.” Amen!  This quote summarizes the problem in a nutshell.

I have been critical of Science as a follower, not a leader, on reproducible research. However, their publication of Couzin-Frankel's report goes a long way to earning forgiveness.

References

Jennifer Couzin-Frankel, 2013: When mice mislead. Science, 342: 922-925.

John P.A. Ioannidis, 2005: Why most published research findings are false. PLoS Medicine, 2 (8), e124: 696-701.