A review of "Research Methods for Science," by Michael Marder. Part 2. Randomization.

In this post I continue my review of Michael Marder’s book, Research Methods for Science (Cambridge University Press, 2009).  I want to dwell here on the concept of randomization, which has many applications in the design of experiments (Peirce & Jastrow, 1885).  For instance, random assignment of subjects to treatments, in a controlled clinical trial, along with double-blinding, helps to rule out competing explanations of the trial outcomes, such as confounding variables and self-selection (which can haunt non-randomized studies).  When there are no systematic differences between treatment groups other than the intervention being evaluated, the results can only be causally linked to either the intervention or to chance.  Similarly in studies where histological samples or medical imaging files need to be scored or segmented by human raters, randomized and blinded presentation of the samples or images should be done to reduce potential systematic errors.  Randomization and blinding together help secure the attribution of causality by ruling out systematic differences between treatment groups other than the intervention under study.

Randomization is never discussed in Marder’s book, despite its hallmark status in comparative experiments, which earns discussion in potentially competing books such as Wilson (1952/1990), Valiela (2009), and Srinagesh (2006).  Marder’s omission is particularly glaring since he promises to show us procedures that avoid bias (p. 4) and later discusses clinical trials (p. 40), where randomization is absolutely essential.  The author does discuss random sampling, in the context of surveys (Sec. 2.3.3), but only hints at the limitations of convenience sampling, the most common type of non-random sampling.  Moreover, he writes that “One way to obtain random lists of student names is to type two-letter sequences into a university directory search engine” (p. 43).  On the contrary, such an exercise is an illustration of haphazard, not random sampling.  The founder of the statistical theory of experimental design, R.A. Fisher, specifically ruled out arbitrary human choice as a source of randomness in his classic 1935 monograph, The Design of Experiments (Fisher, 2003, p. 11).  In contemporary science, engineering, and medicine, there is no excuse for avoiding either pseudo-random number generators (readily accessible in spreadsheets or any decent scientific or statistical software program) or physical sources of random numbers (radioactive decays and suchlike) to produce random assignments or random samples (e.g., Hayes, 2001).

Incidentally, another hallmark principle of experimental design, blocking, appears to be beyond the scope of Marder’s book -- presumably because the analysis of variance is stated to be outside its scope (p. vii).  Competing texts by Valiela (2009) and Srinagesh (2006) do include a discussion of blocking, as does any basic statistical text on experimental design.

Marder’s omission of randomization in a book about how to do science is a fatal oversight.  It is a lost opportunity to introduce good habits and careful experimental procedures.  A discussion of what can go wrong without randomization would have provided opportunity to illustrate critical thinking in science.  Randomization is not an advanced concept, and science undergraduates should be expected to know why it is important and how to carry it out. 

Incidentally, last year a Florida Congressman (who has a bachelor’s degree in electrical engineering) sponsored a bill to end the US Census Bureau’s American Community Survey in part because “in the end this is not a scientific survey. It’s a random survey.”  Go figure!

References

 

 R.A. Fisher, 2003:  The Design of Experiments.  First edition published, 1935; eighth edition, 1966.  Reprinted in R.A. Fisher, 2003:  Statistical Methods, Experimental Design, and Scientific Inference (Oxford University Press).

B. Hayes (2001): Randomness as a resource. American Scientist, 89 (4): 300-304.

C.S. Peirce and J. Jastrow, 1885:  On small differences in sensation.  Memoirs of the National Academy of Sciences, 3:  73-83.

K. Srinagesh, 2006:  The Principles of Experimental Research.  Butterworth-Heineman.

I. Valiela, 2009:  Doing Science:  Design, Analysis, and Communication of Scientific Research.  Second edition.  Oxford University Press.

E.B. Wilson, Jr., 1952/1990:  An Introduction to Scientific Research.  McGraw-Hill, reprinted by Dover.

 

A review of "Research Methods for Science," by Michael Marder. Part 1. Overview.

This post is the first in a 5-part series that will review the book, Research Methods for Science, by Michael P. Marder (Cambridge University Press, 2011).  This book is designed to accompany an innovative undergraduate science course developed at the University of Texas at Austin. The course is intended as an introduction to scientific research, and students are required to formulate their own research questions, do their own experiments and data collection, and present their findings orally and in writing. The course was originally intended for future science teachers, but according to the author, an accomplished physicist, it has grown to include a broad range of students who want an introduction to research.  

Such a course (and accompanying book) could be tremendously useful. Most science majors spend their undergraduate years working through canned problem sets and laboratory exercises. Often they must wait until a senior honors project or even graduate school before they have a chance to formulate research questions of their own. Why not give students a hands-on taste of research very early in their education?  Like the even more ambitious curricular approach taken at the Olin College of Engineering, innovations like this are badly needed to make science education relevant and exciting.  Therefore it is with great regret that I believe the book under review may do as much harm as good, as we’ll see in later posts.  Here, I will just provide an overview of the book and set the stage for later commentary.

The book’s first, introductory chapter is titled “Curiosity and Research.” It examines different types of scientific questions, and defines several modes of scientific research. The latter include hypothesis-driven research, measuring a value or a function/relationship, constructing a theoretical model, observational and exploratory research, and research to improve a product or process (industrial and applied research). The second chapter, “Overview of Experimental Analysis and Design” focuses on hypothesis-driven research, and motivates the discussion with three case studies: comparing the motion of a ball on a ramp with and without lubrication, comparing the lengths of spines on fish from two different lakes, and flipping a coin to evaluate its fairness. The chapter includes extensive discussion of measurement error and how to cope with it. It also discusses laboratory safety as well as ethical issues when using animal and human subjects in research.  The third chapter, “Statistics,” is a primer on elementary statistical analysis. After introducing basic descriptive statistics, it discusses probability distributions, focusing on the binomial and normal distributions, and states the central limit theorem. Hypothesis testing, p-values, confidence intervals, and the Z-, t-, and chi-square tests are all introduced. (Analysis of variance is considered beyond the scope of the text.) The fourth chapter, “Mathematical Models,” has a brief discussion of basic arithmetic, an even briefer discussion of algebra, a discussion of functions, and a qualitative introduction to calculus and differential equations. Order of magnitude estimation and dimensional analysis are covered, as are simple linear regression, correlation, Fourier analysis, and iterative maps (as an example of dynamical systems – a simple model of population growth is analyzed). The final chapter, “Scientific Information,” focuses on writing both proposals for and reports of scientific research, giving presentations, and preparing figures (including data graphics). Also discussed are literature search methods and hints on reading scientific literature productively.

Each chapter has a set of assignments and a reference list. There are four appendices: one on the use of spreadsheets (focusing on Microsoft Excel), a second with an excerpt from one of Galileo's classic dialogues, a third on laboratory safety, and a fourth on grading rubrics for students asked to grade each other’s work. 

I note here that Fig. 3.5 (p. 69) is incorrect:  it is a repeat of Fig. 3.1 (p. 59) but with a different caption.  The caption seems to be correct but the wrong figure inserted.

The author and his colleagues should be commended for devising the course, and offering the text for possible wider use.  Unfortunately, as I will discuss in future posts, the book misses many opportunities to teach critical thinking and good scientific habits, and in some cases reinforces bad habits commonly shared in the scientific community.  Here I will limit my comments to some thoughts on pedagogy, although not being a science educator I will keep these limited. In general I agree that a course like this needs to devote more time to hands-on work and less time to lectures and readings. However I also think much can be gained by having students read and critique a few accounts of well-designed and executed experiments, and just as importantly, those of flawed studies that still made it to publication. (Perhaps the latter should be selected from the works of deceased authors, to avoid irritating the living.) A student can hardly be expected to know what a sound scientific experiment looks like unless he or she has seen a few examples and counterexamples (case studies).

The next part of this series will discuss the single most devastating omission from the book:  randomization.  Two posts will follow to discuss various statistical deficiencies of the book, and a final post will cover a miscellany of other issues with the text.

Lies, Damned Lies, and Medical Science

In the last post I mentioned the epidemiologist John Ioannidis.  I stumbled upon an excellent article about him and his team by David H. Freedman at the Atlantic, Lies, Damned Lies, and Medical Science.  It was written in 2010, and since then Ioannidis has continued to publish critical meta-analsyes, most recently Button et al. (2013).  The first author, Kate Button, has a commentary on the paper at the Guardian, where she vividly describes the research literature as being littered with "bogus or misleading results."

References

Button, K.S., Ioannidis, J.P.A., Mokrysz, C., Nosek, B.A., Flint, J., Robbinson, E.S.J., and Munafo, M.R. (2013):  Power failure:  why small sample size undermines the reliability of neuroscience.  Nature Reviews Neuroscience, 14:  365-376.

NIH ponders reproducible research

Last week, Meredith Wadman at Nature reported that this summer, the NIH has been considering options for concrete steps to promote reproducible research.  This top-down approach is contrasted in the article with discussions of private sector efforts in the same direction, characterized by Wadman as a bottom-up approach. She also summarizes some of the more prominent recent studies of non-reproducible research, although she neglects to cite any of the work of John Ioannidis and collaborators.

The ideas discussed in the article are intriguing, and go part way to addressing some of the issues. I will withhold comment on specifics until the NIH has made public its conclusions.  Nonetheless I applaud the fact that they take the issue seriously and are considering action.