Journals unite for reproducibility: DTLR is pleased!

Today Nature and Science posted joint editorials (here and here) endorsing a Proposed Principles and Guidelines for Reporting Preclinical Research, posted at the U.S. National Institutes of Health.  It is the product of a June, 2014, workshop sponsored by NIH and the two journals, and endorsed by over 30 other biomedical journals.  It provides a bare minimum list of criteria for good reporting practices of animal experiments in biomedical research.  The list is less detailed than one given by Landis et al. (2012) which they cite.

DTLR joins in endorsing the proposed principles and congratulates all the participants for taking a major step forward in promoting reproducible research.  It reflects a discipline-wide concern about the metastasis of non reproducible research across the spectrum of journals, as abundant evidence has made clear in recent years.  The proposed principles emphasize study design issues such as randomization, blinding, sample size, and appropriate replication.  Appropriately, such issues receive more space than analysis.  (The use of the term "inclusion/exclusion criteria" is a little confusing here - in clinical research, this refers to patient enrolment criteria; but the authors here seem to use it to refer to selective reporting and data omission, certainly an important issue, but one I would have found other language to describe.)  The sharing of data sets and the full disclosure of biological reagents are also welcome features.  In general, materials and methods sections of papers really should be expanded such that an independent laboratory could reproduce the experiment and expect to obtain similar results.

DTLR does not believe that the proposed principles go far enough, however.  For instance, the section on statistics requires a disclosure of "the statistical test used"; the fact that the emphasis here is on a statistical test rather than an estimation procedure is a serious oversight, in my view.  The reporting of confidence intervals instead of tests provides a sense of magnitude and direction that is lacking in a p-value, allowing evaluation of both clinical and statistical significance.  A statistical test outcome only communicates statistical significance. A confidence interval implicitly reports a test result when the confidence limits are compared with zero (for a conventional null hypothesis test).  Other types of estimation (tolerance intervals, prediction intervals) may be more appropriate in some situations.

DTLR is glad to see a growing consensus in the scientific community that nonreproducible research is corrosive and must be reduced.  This is a terrific step forward, but just one step.  Now, individual laboratories must take these guidelines to heart and use them to improve the design, execution, analysis, and reporting of their studies. 

References

Nature, vol. 515, p. 7 (2014).

Science, vol. 346, p. 679 (2014).

S. C. Landis, et al., 2012:  A call for transparent reporting to optimize the predictive value of preclinical research.  Nature, 490:  187-191.