A review of "Research Methods for Science," by Michael Marder. Part 5. Miscellany.

In this post I conclude my review of Michael Marder’s book, Research Methods for Science (Cambridge University Press, 2009).  I collect here a miscellany of other thoughts about the book.

Study designs

In chapter 1, Marder makes a distinction between experimental and observational and exploratory studies.  However, he never quite gets around to outlining the strengths and weaknesses of these three approaches, nor more fundamentally the use of retrospective vs. prospective data.  He also does not compare and contrast specific types of widely-used study designs, for instance, a parallel-group design vs. a cross-over design, or in epidemiology, a cohort study vs. a cross-sectional study.  In my view, such a discussion is not necessarily needed for its own sake alone, but as good training for the critical thinking skills for study design.  (A side effect of teaching this material is that students would become better consumers of medical news.)  Unfortunately, I continue to encounter scientists with doctoral level degrees who seem to be lacking awareness of how some study designs make for weaker conclusions than others.  Marder notes the importance of cause and effect in Sec. 2.1.1, but never really teaches how study design helps secure the attribution of causality once the study is completed. He doesn't even introduce the correlation vs. causation fallacy, one that is frequently committed in the scientific literature. These are colossal lost opportunities to teach critical thinking in the book.

Internal and external validity

Another key but missing concept comes from the social sciences, where often a distinction is made between internal validity and external validity. Internal validity refers to whether the study was designed properly and allows for the attribution of causality. A lack of randomization, blinding, concurrent control group, and so on, would threaten internal validity. External validity, on the other hand, refers to how broadly the study's findings can be generalized. A clinical trial is usually based on a convenience sample, filtered by the inclusion/exclusion criteria. The generalization of its results to a larger population is an inherently non-statistical judgment about how representative the trial subjects are of some larger population of interest. Marder does not discuss these issues at all.

Scientific communication

Entire books have been written on scientific communication, and a concise text like Marder's cannot be expected to cover this important topic in depth. Sadly few science students will ever take a formal course on scientific writing. I won't dwell here on the issue, but one of the best brief guides to writing an abstract (in my view) is Sec. IIB of the AIP Style Manual (1997). The relevant passage is less than a page long, and I would simply provide it as a handout to students. Marder covers abstracts in Sec. 5.3.2, and rightly suggests that the abstract “should probably be the very last thing you write” (p. 154). However, I would have added the first sentence of the AIP Style Manual passage: “The primary purpose of the abstract is to help prospective readers decide whether to read the rest of your paper” (p. 5). This theme is expanded upon in Alley (1996), and it really helps to concentrate the mind when writing an abstract. As a referee and a reader, I've run into too many poorly written abstracts, including some written by journal editors themselves!

Marder's discussion of scientific presentations (Sec. 5.5) begins with a description of the kinds and frequency of talks given by scientists. Unfortunately, his description applies mainly to academic scientists, and does not adequately reflect the experience of industrial or government scientists, for whom the topic is equally important. For those of us in the latter categories, a presentation (as opposed to a written report or publication) may often be the deliverable that most influences decision makers, both within and beyond a research organization. A popular technique, not mentioned by Marder, is to conclude a talk with up to three take-away message(s).

Data graphics

Marder prefers that statistical graphics not show all the raw data, but only display the mean and error bars (Sec. 5.4.2). Plots of the raw data “have the defect of providing a little too much information for rapid understanding” (p. 163). A plot displaying only the mean and standard deviation “does not show all the work you did by performing many different trials” but “The point of a scientific publication is not to explain to everyone how much work the scientist did, but to convey the results. This is the most compact way” (p. 163). I strongly disagree with making such a blanket statement. The choice of what to display is highly context dependent, in my view. Moreover, there are ways to display the raw data (overlaid with boxplots using jittering or beeswarm, or using density plots, etc.) that better tell the story of the data than the straw man example he gives in Fig. 5.2. Statistical graphics is a modern, advanced discipline, informed by an understanding of human visual perception. With 21st century software, there is often no need to hide the actual data behind error bars, as Marder advocates.  Incidentally, users of Marder’s book and anyone else interested in data graphics should consult the paper by Cumming et al. (2007) which provides an excellent discussion of error bars and how they may be misused.  Marder doesn’t fall into any of the traps described by Cumming et al., but he surely fails to warn readers about them.

Literature search

Marder's discussion of literature search (Sec. 5.6) provides good advice to start with ISI's Web of Knowledge. Although I agree with the author here, I work in an organization that does not provide me with direct access to Web of Knowledge. Instead, I am provided with Elsevier's SCOPUS, which doesn't even appear on Marder's otherwise well curated list of literature databases (Table 5.3, p. 171).  (Some in the scientific community oppose Elsevier's journal pricing policies, which have led many to boycott Elsevier products.)

Concluding thoughts

If I wanted to give a budding scientist a nuts-and-bolts guide to both doing research and critically evaluating the research of others, I would not turn to this book.  However, I don’t know if the book I’m really looking for exists at all.  Readers, do you have any suggestions?

References

M. Alley (1996): The Craft of Scientific Writing, 3d ed. Springer-Verlag.

American Institute of Physics (1997): AIP Style Manual, 4^th ed.   

G. Cumming, F. Fidler, and D.L. Vaux (2007): Error bars in experimental biology. J. Cell Biol., 177: 7-11.