Back in January, Science magazine
featured a story by Jennifer Couzin-Frankel about a pair of dissident
medical scientists, Arturo Casadevall and Ferric Fang. They're
convinced that science today is very unhealthy, and they have done a
number of data-driven studies on the infrastructure of scientific
research to diagnose various problems with the enterprise. They've
looked at peer review, retraction rates, funding mechanisms, and the
incentive system for scientists. Initially I was not familiar with
their work, so I began by reading their two editorials in Infection
and Immunity, here and here, alluded to in their Huffington Post blog post from 2012. (They also revisit these issues in a post on Project
Syndicate from this past summer.) Their intent is to spark
conversation rather than provide definitive solutions, so I will
accept the invitation to discuss their findings. They write, “What
we propose is nothing less than a comprehensive reform of scientific
methodology and culture” and I certainly agree that such a dramatic
overhaul is needed.
This blog, DTLR, has been particularly
concerned about non-reproducible research. Casadevall and Fang do
not address this particular issue directly. Rather, they tackle some
of the underlying pressures on scientists due to the incentive system
they current face. Their diagnosis of the problems has the ring of
truth, and I refer readers to their papers; I won't rehash their work
here. Rather, I want to comment on their proposed solutions.
Addressing the issues they've identified is likely to reduce the
incidence of non-reproducible research that I've written about on
this blog.
First, the authors propose reforming
the reward system for scientists. This includes eliminating the
priority rule, which gives credit to the first to publish. The
authors recognize the value of competition, but they want to
introduce complementary reward mechanisms for collaborative research.
No specific details are given, so the idea needs to be fleshed out.
Nonetheless I agree on the principle. They also want to replace easy
“surrogate methods of quality, such as grant dollars, bibliometric
analysis, and journal impact factor” with “careful peer
evaluations of scientific quality”. Of course, this is easier said
that done. In principle I agree here too, but we need to see a
detailed, specific proposal on how this would be done. Arguments
about the flaws of impact factors, h-indices, and so on are a
perennial favorite in the scientific community, so doubtless there
are many ideas out there.
Next, the authors talk about
re-embracing philosophy. It sounds to me that they want to add one
or more philosophy courses to the curriculum for science students.
Practically, this would be difficult, as the curriculum is pretty
stocked full. Moreover, I think that if students are forced to take
such a course, many may not take it seriously and others may resent
it. I happen to be one of those who would welcome such a course (and
I did take Philosophy of Science in college), but I would hesitate to
impose my values on others. Also, I think some students might rather
read a book or two rather than take a full-blown, for-credit course
on the subject. Thus, I would make such a course available to every
student, but keep it optional rather than required. (In my case,
I've continued to read bits and pieces of philosophy throughout my
career in science.)
Next, they call for enhanced training
in probability and statistics. They recognize the value of
statistics in the “design, execution, and interpretation of many
scientific experiments.” I would certainly applause, with a
caveat. Training should be focused on statistical thinking, with
statistical methodology taking a back seat. Too often the training
students do get on statistics is focused on methods and software,
rather than critical thinking. The misuse of statistical methods,
abundant in the literature, is the result. Many scientists, and many
statisticians, are simply not qualified to teach statistical
thinking. My book review of Marder (2009) on this blog touches on
many issues in this regard, and I plan to tackle other specific cases
in future posts. Also in a future post, or more likely
in another forum altogether, I will expand on the discussion of
statistical thinking vs. statistical methodology in relation to
scientific training and practice.
Next, the authors call for developing
checklists for reducing methodological errors. This is a superb
suggestion, and they provide an example for one case, “observation
in which a stimulus elicits an effect”. The example is well
thought out. Earlier on this blog I noted a new checklist introduced
by Nature for encouraging reproducible research. These are welcome
steps forward; more progress is needed.
The authors then turn to structural
reforms. They call for more public funding for science and an
increase in the number and diversity of new scientists. They find
that in recent years, directed research has overshadowed
investigator-initiated research in NIH funding. Here I would part
company with the authors. I do agree that working on increasing the
diversity of the scientific community is important. I cannot agree
that increasing the absolute size of the community, either in number
of scientists or amount of funding, is a realistic or desirable goal.
Frankly, until the scientific community cleans up its own act, by
dramatically restructuring the incentive system so that reproducible
research is rewarded and non-reproducible research is punished, I
would vote to reduce science funding even further, and I would
discourage young people from considering science careers. This is a
very harsh stance, but I don't think any stance less extreme will
convey my level of anger and dismay. As a practical matter, in the
United States is is politically unlikely that funding for science
will increase when funding for many other worthy societal goals is
necessarily decreasing.
Finally, the authors call for reform in
the regulatory burden on scientists, restrictions on laboratory size,
and a “scientific study of science.” I do not dwell in the
academic trenches of science, so I will only briefly discuss the last
point. They discuss trying to discover the optimal number of
scientists in society, optimal research group size, optimal time for
scientific training, etc. I would encourage study of all of these
questions, but I would not expect blanket “optimal” answers to
result. The system must allow for slack and variability. Such
research can only provide guidance, not mandatory and immutable
rules. In fact, I would expect studies to address some of the
questions they pose to end up completely inconclusive.
Nonetheless, Casadevall and Fang
deserve much praise for asking very tough questions and calling for
a dramatic reform of the science community's infrastructure. Although
I fail to agree with all of their positions, on balance I believe
they are on target, and the rest of us should rally around them.
Science needs more internal critics like them and like John A.P.
Ioannidis (whom I've written about in an earlier post). All of these
folks go out on a limb to make controversial observations about how
science functions and how it goes wrong, as well as proposing
solutions. We all need to follow their lead.
References
Arturo Casadevall and Ferric C. Fang,
2012: Reforming science: methodological and cultural reforms.
Infect. Immun., 80 (3): 891-896.
Jennifer Couzin-Frankel, 2013: Shaking
up science. Two journal editors take a hard look at honesty in
science and question the ethos of their profession. Science, 339:
386-389.
Ferric C. Fang and Arturo Casadevall,
2012: Reforming science: structural reforms. Infect. Immun., 80
(3): 897-901.
No comments:
Post a Comment