Reproducible research in computational science and engineering

The Diffusion Tensor Literary Review (DTLR) endorses the principles of reproducible research outlined in the final report of the ICERM Workshop on Reproducibility in Computational and Experimental Mathematics.  A summary of the findings was published last month by Stodden, Borwein, and Bailey (2013).

The message seems to be that computational science has failed to maintain standards of reproducibility expected in theoretical and experimental science, let alone good software engineering principles.  This has led to a "credibility crisis".  The workshop participants agreed on three principles, which I quote verbatim from Stodden, et al. (2013):

1. "It is important to promote a culture change that will integrate computational reproducibility into the research process."
2. "Journals, funding agencies, and employers should support this culture change."
3. "Reproducible research practices and the use of appropriate tools should be taught as standard operating procedure in relation to computational aspects of research."

Read the final report of the workshop:  it is surprisingly direct, candid, and eloquent for a committee-produced document.  I have not studied the associated wiki in detail, but the report establishes the big picture.  We need to get the scientific/engineering community on board with these basic principles.  It is difficult to understate that a major cultural change is indeed called for, as the current infrastructure actively discourages scientists from implementing principles of reproducible research.

Personally, as a taxpayer and as a scientist, I find the current state of affairs disgraceful.  Scientists and engineers typically spend other people's money (usually at the expense of the taxpayer) to do their work.  Ensuring that results are reproducible seems to be a minimum expectation for publication, yet in computational science and engineering it is not.  Often scientists cannot even reproduce their own work, let alone the work of others.  We need to be better caretakers of the limited amount of money that society collectively has to spend on science and engineering.

Reproducible research is a theme I plan to return to in future posts on DTLR.  Watch this space for more.

 Reference

V. Stodden, J. Borwein, and D. H. Bailey, 2013:  "Setting the default to reproducible" in computational science research.  SIAM News, 46 (5):  4-6.