A review of "Research Methods for Science," by Michael Marder. Part 4. More about statistics.

In this post I continue my review of Michael Marder’s book, Research Methods for Science (Cambridge University Press, 2009).  In my last post I began to take issue with the statistical thinking illustrated in the book; I will continue doing so here.

The coin flipping problem

As mentioned in Part 1 of this review, coin flipping is one of Marder's main examples for illustrating hypothesis driven research. However, his analysis of coin flipping data is inconsistent, sometimes treating the resulting data (correctly) as binary and other times (incorrectly) applying statistical methods intended for floating point data. Marder pays lip service to matching the statistical methods to the data type, most emphatically during his discussion of the chi-square test (p. 101), but doesn't follow this principle when dealing with his coin flipping data.

Let me be specific here. The coin flipping experiment is introduced in Sec. 2.1, and he represents heads with 1's and tails with 0's (p. 23). In Sec. 3.3 he correctly uses a binomial probability model for such data. Note that in this section, he states that “Assigning 0 and 1 to heads and tails is an arbitrary choice; any two integers would do” (p. 63). However, in most other places in the book he takes these integers literally, by calculating means and standard deviations on such data, interpreting these quantities, and even carrying out a Z test (pp. 28-29, 71-73, 88-89)!  None of these arithmetical operations is meaningful for integer data, let alone binary data merely coded as integers.  At the very least, Marder's interpretation of such calculations depends very much on the choice of special integers, 0 and 1. His approach does not generalize to multinomial non-numeric data, and students must be taught not to assign arbitrary integers to categorical data if the analysis will take these integers literally (e.g., by applying methods designed for floating point numbers).  I would argue that much of Marder’s analysis of the coin flipping problem is technically incorrect, but the wider point is that his treatment illustrates and blesses poor practices and bad habits.

Correlation and regression

These topics are discussed in Sec. 4.3. Marder describes the least squares linear regression line as the best fit line, but such a description is not always valid. The calculations he describes in this section are based on conditioning on the dependent variable. When this is appropriate, we can call the resulting line a best fit line. However, it is common in science to encounter cases where such conditioning is not appropriate, such as when the scientific interest is in estimating the relation between the two variables rather than predicting one, given knowledge of the other. Moreover, in cases where both variables are random and/or measured with measurement error, the least squares line may provide a biased estimate of the slope. Better solutions in these situations have been around for years (e.g., orthogonal distance regression, geometric mean regression, measurement error regression), but are not typically discussed in introductory books (including Marder's). Consequently many scientists proceed with simple linear regression in cases where it is not appropriate, because they don't know any other way.

Marder also describes the correlation coefficient as “a measure of how well or poorly the points...fall on a straight line” (p. 127). This is true but not the whole truth. The correlation coefficient is also sensitive to the slope of the fitted line, and thus it is not a pure measure of the clustering of data about the best fit line (e.g., Loh, 1987).

Sensitivity and specificity

Sec. 3.6.6, on Type 1 errors, ends with an example: mammogram error rates. He describes the false positive and false negative rates of mammograms, with a little bit of context. However, the discussion is very cursory and fails to acknowledge an important distinction: the false positive and false negative rates are conditional on knowing the true status of the disease. When diagnostic results are reported to an individual patient, however, the probabilities that the patient is actually positive or negative for the disease, given the reported diagnosis, can be quite different from the diagnostic test's sensitivity and specificity, since the former probabilities also depend on the prevalence of the disease. These quantities (positive predictive value and negative predictive value) are conditioned in the opposite direction from the sensitivity and specificity discussed by Marder, and the two sets of error rates are frequently confused with each other, even by medical professionals. (In the legal profession, a somewhat similar confusion involving the direction of conditioning is known as the “prosecutor's fallacy.”) The routine use of mammograms in certain age groups has been a politically and medically controversial topic in recent years, and many readers of the book know or will come to know a woman receiving a positive diagnosis from a mammogram. The cursory and incomplete discussion of mammogram error rates in Marder's text, I fear, may be harmful to readers if left without further explanation. Either he should omit the example, or give a fully nuanced discussion of it, as can be found for instance in Gigerenzer (2002). 

Precision and accuracy

Marder's discussion of precision and accuracy (Sec. 2.2.3) is an example of muddled writing. He never actually provides definitions for the two terms, and seems to conflate precision with numerical precision and significant digits (a distinct topic otherwise absent in the book). Accuracy refers to the systematic error of a measurement process; it is quantified by the deviation between a true value (for instance, of a standard sample assigned a reference value) and the mean of replicated measured values. Precision is the degree of scatter of replicated measured values about their mean, regardless of accuracy. Measurements can be accurate and not precise, and vice versa. Examples of clearly written discussions of precision and accuracy may be found in Dunn (2010, Sec. 7.5) and at greater length in Mandel (1984, chapter 6).

Model assumptions

In the discussion of U.S. population growth models (Sec. 4.6.1), an opportunity is lost to illustrate checking the plausibility of model assumptions. Although Marder concedes that his models' prediction “is only as good as the assumptions that led to” his model (p. 140), he makes no attempt to evaluate the plausibility of these assumptions. The model assumes that there is a fixed maximum population and a fixed growth rate. The growth rate is assigned the value of a historical average (for 1950-2000), and the maximum population is arbitrarily set to 6.4 billion. The author's discussion seeks mainly to illustrate how to simulate an iterated map model in spreadsheet software, but he could have also given students some guidance on comparing a model's forecasts with, say, historical data or data from other countries. Such comparisons are an essential component of model validation, a process commonly discussed in engineering. However, in "pure" science model validation is rarely discussed in any formal way, except in certain fields where forecasting is of central interest (e.g., weather and climate).

Error propagation

In Sec. 2.2.4, Marder discusses the rules for propagation of error when calculating quantities consisting of arithmetic combinations of other quantities measured with error. The rule for adding two quantities (their uncertainties add in quadrature) is justified by citing a probabilistic argument given later in the book (p. 80), which shows that this is true for standard deviations when adding two independent random variables. This is a subtle sleight of hand. The rules for propagation of error apply equally to deterministic uncertainties, as justified by approximation theory (namely, Taylor approximation) as shown by Taylor (1997). It is reassuring that the approximation theoretic and probabilistic rules for propagating error are consistent. However, an opportunity is lost to distinguish between a deterministic uncertainty, such as the finite resolution of a measurement system, which can be applied to a single measurement, and a probabilistic uncertainty, which can only be evaluated by replicated, independent measurements. Incidentally, the latter is preferred, since the uncertainty of a single measurement based on finite resolution is almost always an underestimate of the full uncertainty of the measurement process, which can only be evaluated by replicated, independent measurements, preferably carried out on multiple days by multiple technicians, under varying conditions.  The uncertainty of a measurement instrument is always less than the uncertainty of a measurement process as a whole -- this is another arch principle of statistical thinking often lost on physicists (of which I am one).  Ironically, it was the great physicist-turned-statistician W. Edwards Deming, among others, who promoted a process-focused view of variability.  (I hope to explore this issue in greater depth in a future post.)

The next and final post of this series will collect some miscellaneous criticisms of the book.

References

P.F. Dunn (2010):  Measurement and Data Analysis for Engineering and Science, 2d ed. CRC Press.

G. Gigerenzer (2002):  Calculated Risks: How to Know When Numbers Deceive You. Simon & Schuster.

W.-Y. Loh (1987):  Does the correlation coefficient really measure the degree of clustering around a line? J. Educ. Stat., 12: 235-239.

J. Mandel (1984):  The Statistical Analysis of Experimental Data, corrected reprint. Dover.

J.R. Taylor (1997):  An Introduction to Error Analysis: The Study of Uncertainties in Physical Measurements, 2d ed. University Science Books.

A review of "Research Methods for Science," by Michael Marder. Part 3. Statistical thinking.

In this post I continue my review of Michael Marder’s book, Research Methods for Science (Cambridge University Press, 2009).  Last time I discussed the author’s failure to discuss randomization in experimental design. A second lost opportunity is the lack of a systematic discussion of sources of variability.  Although chapter 3 is dedicated to statistics, chapter 2 introduces a number of statistical concepts and methods, in the context of “error analysis.” The author spends a lot of time in the chapter discussing measurement error (both random and systematic), which is a good thing. However, he fails to acknowledge that other sources of variability (not just measurement error) need to be accounted for in experiments in the life and social sciences. Biologists use the terms “biological variability” and “technical variability” to distinguish between natural biological variation in a given population and measurement error, respectively. Chapter 2's example of the fish spine length measurements presents a good (but lost) opportunity to make this important distinction. Statistical analysis in this example is required not so much due to measurement error (which is not assessed at all in the study as Marder describes it) but rather due to the natural variation of fish spine lengths in the two lakes.  Partitioning the sources of variability is one of the most important concepts in applied statistics, and although the machinery of analysis of variance is needed to fully cope with it, the author could have at least introduced it conceptually. Measurement error in the fish spine study would have been assessed by taking multiple, independent measurements on each fish, and its extent would likely be much lower than the variation between fish.  A study incorporating such measurements also presents an example of structured data that cannot all be considered independent and identically distributed.   Structured data is too ubiquitous in science and ordinary life to leave unmentioned. 
 
A third lost opportunity is the distinction between statistical and practical/clinical significance.  The author pays lip service to this distinction (p. 54) but does not walk the walk.  Earlier in the book (p. 18) he presents, as an example of hypothesis driven research, the question of whether one aspirin cures at a faster rate than another.  An even earlier example is “finding which of two medical treatments is more effective” (p. 4). As John Tukey (1991) would have argued, the cure rates of the two aspirins will always differ at some decimal place. The more important question is by how much do the cure rates differ, and how precisely do we know it? The answers are provided by point and interval estimation, not hypothesis testing (e.g., Gardner & Altman, 1986). (In fact, reporting the confidence interval tacitly implies a hypothesis test: if the confidence interval of the difference in cure rates overlaps the null hypothesis value, typically zero, statistical significance was not demonstrated.) In the author’s defense, the obsession with hypothesis testing at the expense of estimation is rampant in the scientific community, and statisticians themselves are partially to blame for this misplaced emphasis. The resulting attitude can lead to perverse consequences. A practically or clinically negligible difference between medical treatments can always be found statistically significant when a sufficiently large number of patients are enrolled in a clinical trial. This has led to oncology trials of treatments with small therapeutic effects (and non-negligibly harmful side effects) in large studies in order to chase statistical significance (Horrobin, 2003). This is an abuse of the altruism asked of patients in these trials, who must suffer before they die, in order to secure knowledge of a statistically significant but clinically questionable outcome. 

The examples of statistical results reported in Sec. 5.3.5 (p. 159), are dominated by hypothesis testing; no confidence intervals for quantities of scientific interest are provided.  In the lichen example, Marder does provide standard errors for meaningful quantities (lichen sizes from two regions), but not a confidence interval for their difference (which appears to be the pivotal question of the study; only a p-value is given.) In the stickleback example, he again provides estimates of scientifically meaningful quantities, but without any uncertainty statements attached to these estimates. In both of these examples, p-values are presented with more than two decimal place precision: such numerical precision is almost never meaningful.  

Another example of the weakness of the hypothesis testing approach is illustrated by Marder’s bottle rocket height example, discussed in Sec. 3.8, on the chi-square goodness of fit test. Here, a parabolic curve is fitted to trajectory data, and the chi-square test is used to evaluate the quality of the curve fit. This is a common practice, but questionable when unaccompanied by additional work. Goodness of fit should not be boiled down to a single number, the p-value of a statistical test. Quality of fit may be good in one region of the curve but poor in another—plotting the data (Fig. 3.13) helps to evaluate where and how the fit may be poor. The root mean square error (RMSE) of the curve fit provides a quantitative estimate of how scattered the data are around the fitted curve. An estimate like the RMSE can help evaluate whether goodness of fit may be practically acceptable, regardless of statistical significance, which is driven as much by sample size as magnitude of error.  The chi-square p-value alone cannot address practical significance.
 
In Sec. 2.1.1, Marder discusses null and alternative hypotheses.  However, he fails to make a distinction between conventional null hypotheses of no effect (innocent until proven guilty) and equivalence hypotheses, where one is trying to demonstrate that two effects are (by pre-specified criteria) equivalent, whereas the default position is that they are not.  (Such hypotheses are useful for instance in research on generic drugs.)  Occasionally I’ve seen scientists make use of conventional null hypothesis testing where equivalence testing is more appropriate, usually because they know no other way.  Marder has one example which resembles the issues raised in the equivalence hypothesis: in Table 2.1 (p. 19), he lists an alternative hypothesis that “A Toyota Camry weighs exactly 1000 kg” with the null hypothesis that it does not. This pair of hypotheses is structured like a “one sample” version of an equivalence hypothesis, but he does not comment further about its alternative structure, compared to the more traditional hypothesis pairs in Table 2.1.  In Table 1.1 (p. 5), he says that the hypothesis that a Toyota Camry weighs exactly 1000 kg is “silly if left as a hypothesis. There is no reason that the weight of a car should come out to such a neat round number.” He doesn't seem to realize that the same “silliness” exists in “two sample” problems – as mentioned above, two medical treatments are always different to some decimal place, possibly negligibly so (Tukey, 1991).

Addressing the complaints outlined in this and the previous posts would not require lengthening the book by much.  Randomization could be dealt in less than a page in the main text, with another page in the spreadsheet appendix showing how one could actually carry it out.  Sources of variation could be addressed in a single paragraph in the fish spine length example, with a reference to a more advanced text provided.  Addressing practical/clinical significance vis-a-vis hypothesis testing requires rewriting a number of passages, but without making them lengthier by much.  In short, it should have been easy to make Research Methods for Science a much better book while keeping it short.  I will continue to examine the book's deficiencies in the next two posts.

References

M.J. Gardner and D. G. Altman (1986): Confidence intervals rather than p values: estimation rather than hypothesis testing. British Medical Journal, 292: 746-750.

D.F. Horrobin (2003): Are large clinical trials in rapidly lethal diseases usually unethical? Lancet, 361: 695-697.  [I hesitate to cite work of a figure as controversial as Horrobin, but I think the ideas expressed in this particular paper, published a few months before his death from cancer, may actually have some merit.]

J.W. Tukey (1991): The philosophy of multiple comparisons. Statistical Science, 6: 100-116.