Data stewardship

In last week's issue of Eos, Karen Simmons (2013) writes about data stewardship and her experience of attempting to recover and migrate data from long completed NASA space probe missions from the 1970s and 1980s.  Some of the data (on 7-track tapes and punch cards) was about to be discarded, as the storage facility was being shut down.  Much of the valuable metadata required to interpret the raw data is buried in mission documents that may be lost as personnel retire, change jobs, or even change offices.  The article is well worth reading and, though she doesn't mention it, the issues she raises are closely related to reproducible research.  Although the difficulties described are particularly extreme (the casual discarding of data from expensive, taxpayer-funded space probes, each unique in their own way, but long decommissioned), any of us doing experimental or observational science has much to learn from her experience.  Sadly, she finds that "usable science data and metadata from the early days of planetary exploration are now missing from the archives."  Don't let this happen to you!

Reference

K.E. Simmons, 2013:  Lost Science:  Protecting Data Through Improved Archiving.  Eos, Transactions of the American Geophysical Union, 94 (37):  323-324.

The AMS Data Policy Statement: Full and Open Access to Data

The American Meteorological Society currently has a draft statement open to comments by members until Oct. 3, on Full and Open Access to Data.  As a member I submitted the following comment:

"I applaud the principles outlined in this document, and AMS's willingness to take a definite stand on the issues discussed. I am particularly in favor of strong encouragement for academic journals and funding agencies to *require* that data sources be clearly identified and publicly available, unless a justification can be given.  What the policy fails to address, perhaps because it is out of scope, is a more complete endorsement of the principles of reproducible research, which would also require making computer code publicly available, and also specification of software options and settings used, and finally full specification of any data reduction or manipulation procedures carried out between the raw and the analyzed data sets (e.g., filtering, interpolation to convert non-equally spaced time series into equally spaced time seris, etc.)  Thank you for this opportunity to comment."

The Diffusion Tensor Literary Review (DTLR) similarly endorses the draft statement on Full and Open Access to Data. 

Why DTLR?

Some readers may be wondering about the name of this blog, “The Diffusion Tensor Literary Review.” Obviously I am not blogging about diffusion tensors! Some of you know that a diffusion tensor is a mathematical construct used to represent non-isotopic diffusion of (usually) water in some medium, like brain or muscle tissue, and it finds particular application in magnetic resonance imaging. Years ago I did have collaborators who were doing diffusion tensor imaging, but that is not what this blog is about. The blog's name is a little bit of a joke, as I've appropriated some obscure, narrow-sounding techno-babble to describe a blog that I hope will range widely in subject matter.

Actually, not quite. I like the name for deeper reasons. Both the American Physical Society and the American Institute of Physics have in their mission statements (and I paraphrase) the advancement and diffusion of the knowledge of physics. While I don't plan to do any advancing of knowledge here, I surely hope to diffuse some. And why stop at diffusion? I'll try to convect and radiate knowledge too!

And what about “tensor”? Back in college, I was the back-page columnist for the Math Club's newsletter (which had only one issue in its then-incarnation). My column was named “The Tensor.” That too was a joke, for in mathematics the word “column” reminds one of a column vector in matrix algebra, and matrices are kind of like tensors! (Rest assured, the Math Club was highly proficient in bad puns and lame humor.)

Hence, the Diffusion Tensor Literary Review. It is indeed primarily a literary review, in the sense of a journal club and book club, and not in the literal sense of reviewing (fictional) literature. The recent series of posts on Michael Marder's book, Research Methods for Science, is an example of what I hope to do on this blog.

Finally, what about the blog's avatar? It is a Computerworld button that I saw at the Computer History Museum in Mountain View, CA. It reads, “Garbage in, gospel out.” Of course, computer scientists are fond of the opposite saying, “Garbage in, garbage out.” The button stating “Garbage in, gospel out” represents a certain attitude that the output of a computer is gospel. This attitude can be more generally held for the output of statistical analysis, or indeed of any scientific endeavor. I think of “Garbage in, gospel out” as a motto for non-reproducible research, an issue that plagues the biosciences in particular these days. Non-reproducible research, particularly when enabled by the use of statistical methodology (as opposed to statistical thinking), will continue to be a pivotal concern of this blog.

A review of "Research Methods for Science," by Michael Marder. The compilation.

For convenience, I've compiled here links to the five-part series of posts that make up my review of Research Methods for Science, by Michael P. Marder (Cambridge University Press, 2009).

Part 1. Overview.

Part 2. Randomization.

Part 3. Statistical thinking.

Part 4. More about statistics.

Part 5. Miscellany.

A review of "Research Methods for Science," by Michael Marder. Part 5. Miscellany.

In this post I conclude my review of Michael Marder’s book, Research Methods for Science (Cambridge University Press, 2009).  I collect here a miscellany of other thoughts about the book.

Study designs

In chapter 1, Marder makes a distinction between experimental and observational and exploratory studies.  However, he never quite gets around to outlining the strengths and weaknesses of these three approaches, nor more fundamentally the use of retrospective vs. prospective data.  He also does not compare and contrast specific types of widely-used study designs, for instance, a parallel-group design vs. a cross-over design, or in epidemiology, a cohort study vs. a cross-sectional study.  In my view, such a discussion is not necessarily needed for its own sake alone, but as good training for the critical thinking skills for study design.  (A side effect of teaching this material is that students would become better consumers of medical news.)  Unfortunately, I continue to encounter scientists with doctoral level degrees who seem to be lacking awareness of how some study designs make for weaker conclusions than others.  Marder notes the importance of cause and effect in Sec. 2.1.1, but never really teaches how study design helps secure the attribution of causality once the study is completed. He doesn't even introduce the correlation vs. causation fallacy, one that is frequently committed in the scientific literature. These are colossal lost opportunities to teach critical thinking in the book.

Internal and external validity

Another key but missing concept comes from the social sciences, where often a distinction is made between internal validity and external validity. Internal validity refers to whether the study was designed properly and allows for the attribution of causality. A lack of randomization, blinding, concurrent control group, and so on, would threaten internal validity. External validity, on the other hand, refers to how broadly the study's findings can be generalized. A clinical trial is usually based on a convenience sample, filtered by the inclusion/exclusion criteria. The generalization of its results to a larger population is an inherently non-statistical judgment about how representative the trial subjects are of some larger population of interest. Marder does not discuss these issues at all.

Scientific communication

Entire books have been written on scientific communication, and a concise text like Marder's cannot be expected to cover this important topic in depth. Sadly few science students will ever take a formal course on scientific writing. I won't dwell here on the issue, but one of the best brief guides to writing an abstract (in my view) is Sec. IIB of the AIP Style Manual (1997). The relevant passage is less than a page long, and I would simply provide it as a handout to students. Marder covers abstracts in Sec. 5.3.2, and rightly suggests that the abstract “should probably be the very last thing you write” (p. 154). However, I would have added the first sentence of the AIP Style Manual passage: “The primary purpose of the abstract is to help prospective readers decide whether to read the rest of your paper” (p. 5). This theme is expanded upon in Alley (1996), and it really helps to concentrate the mind when writing an abstract. As a referee and a reader, I've run into too many poorly written abstracts, including some written by journal editors themselves!

Marder's discussion of scientific presentations (Sec. 5.5) begins with a description of the kinds and frequency of talks given by scientists. Unfortunately, his description applies mainly to academic scientists, and does not adequately reflect the experience of industrial or government scientists, for whom the topic is equally important. For those of us in the latter categories, a presentation (as opposed to a written report or publication) may often be the deliverable that most influences decision makers, both within and beyond a research organization. A popular technique, not mentioned by Marder, is to conclude a talk with up to three take-away message(s).

Data graphics

Marder prefers that statistical graphics not show all the raw data, but only display the mean and error bars (Sec. 5.4.2). Plots of the raw data “have the defect of providing a little too much information for rapid understanding” (p. 163). A plot displaying only the mean and standard deviation “does not show all the work you did by performing many different trials” but “The point of a scientific publication is not to explain to everyone how much work the scientist did, but to convey the results. This is the most compact way” (p. 163). I strongly disagree with making such a blanket statement. The choice of what to display is highly context dependent, in my view. Moreover, there are ways to display the raw data (overlaid with boxplots using jittering or beeswarm, or using density plots, etc.) that better tell the story of the data than the straw man example he gives in Fig. 5.2. Statistical graphics is a modern, advanced discipline, informed by an understanding of human visual perception. With 21st century software, there is often no need to hide the actual data behind error bars, as Marder advocates.  Incidentally, users of Marder’s book and anyone else interested in data graphics should consult the paper by Cumming et al. (2007) which provides an excellent discussion of error bars and how they may be misused.  Marder doesn’t fall into any of the traps described by Cumming et al., but he surely fails to warn readers about them.

Literature search

Marder's discussion of literature search (Sec. 5.6) provides good advice to start with ISI's Web of Knowledge. Although I agree with the author here, I work in an organization that does not provide me with direct access to Web of Knowledge. Instead, I am provided with Elsevier's SCOPUS, which doesn't even appear on Marder's otherwise well curated list of literature databases (Table 5.3, p. 171).  (Some in the scientific community oppose Elsevier's journal pricing policies, which have led many to boycott Elsevier products.)

Concluding thoughts

If I wanted to give a budding scientist a nuts-and-bolts guide to both doing research and critically evaluating the research of others, I would not turn to this book.  However, I don’t know if the book I’m really looking for exists at all.  Readers, do you have any suggestions?

References

M. Alley (1996): The Craft of Scientific Writing, 3d ed. Springer-Verlag.

American Institute of Physics (1997): AIP Style Manual, 4^th ed.   

G. Cumming, F. Fidler, and D.L. Vaux (2007): Error bars in experimental biology. J. Cell Biol., 177: 7-11.