Communication in Science: Pressures and Predators

The October 4, 2013, issue of Science features a special section, “Communication in Science: Pressures and Predators.” The item receiving the most attention is a sting operation by John Bohannon, who submitted versions of a fake, deliberately flawed article to 304 open access journals. Roughly half of them were actually accepted for publication. It is difficult to separate those journals who did so due to a slipshod editorial process (which is shared by many traditional journal publishers) and those that are deliberately predatory, the topic of Jeffrey Beall's blog.

Bohannon's experiment was criticized by Michael Eisen for not including a control group, that is, a group of traditional, subscription-based journals, in its sample. Philip Moriarty at Physics Focus echoes Eisen's criticisms. Both are fairly hostile to traditional journals for similar reasons. The epidemic of nonreproducible research, discussed in earlier posts on DTLR, serves to illustrate the broken-ness of the peer review system that they speak of. (Ironically, Eisen himself is interviewed in the piece that follows Bohannon's in the Science special feature, about “heretical” publisher Vitek Tracz.)  Eisen and Moriarty are pretty angry at Science for their hypocrisy, but they should have read the rest of the special feature. Jennifer Couzin-Frankel's piece on “The Power of Negative Thinking”, which advocates publication of negative results, is quite blunt about the epidemic of non-reproducible research. For me, Couzin-Frankel's piece is the most important article in the special feature, and I will dedicate a separate post to it shortly.  (Also of great interest is the Policy Forum piece by Diane Harley -- I may write further about that one too.)

In the meantime, the points made by Eisen and Moriarty are well taken. Nonetheless, as Bohannon, Beall, and others have shown, the open access journal movement has opened the floodgates for hundreds of predatory online journals that maintain no standards whatsoever. This surely deserves the widespread coverage that Bohannon's piece has garnered. I only wish that Couzin-Frankel's article had attracted equal scrutiny, along with the recent moves by Nature to raise its level of play on these matters, discussed earlier on DTLR.

It should be disclosed that I once served as a referee for an open access journal from a publisher on Beall's list. Thus I can testify that they (OMICS Group) at least did send out one paper from one journal for review. (Fortunately my review was positive and the paper was published; I do not know what would have happened had I submitted a negative review. I also strongly suspect that I was chosen to referee the paper because the submitting author was asked to provide a list of names of potential reviewers. Many traditional journals do the same.) Some of the other publishers on Beall's list do not even bother with even the appearance of a legitimate review process. I have also had a paper of my own rejected by an open access journal, one not included on Beall's list. The publisher of that journal, Hindawi, also passed Bohannon's test and rejected his phony paper.

Recommended reading

The Special Section in Science contains the following items (as well as a number of sidebar pieces by Jon Cohen and the other authors). Readers' attention was also called to a number of related Perspective and other items appearing in the same issue.

Richard Stone and Barbara Jasny, 2013: Scientific discourse: buckling at the seams. Science, 342: 57.

A cartoon by Randall Munroe (XKCD).

John Bohannon, 2013: Who's afraid of peer review? Science, 342: 60-65. (“A spoof paper concocted by Science reveals little or no scrutiny at many open-access journals”)

Tania Rabesandratana, 2013: The seer of science publishing. Science, 342: 66-67. (“Vitek Tracz was ahead of the pack on open access. Now he wants to rewrite the rules of peer review”)

Jennifer Couzin-Frankel, 2013: The power of negative thinking. Science, 342: 68-69. (“Gaining ground in the ongoing struggle to coax researchers to share negative results”)

David Malakoff, 2013: Hey, you've got to hide your work away. Science, 342: 70-71. (“Debate is simmering over how and when to publish sensitive data”)

Eliot Marshal, 2013: Lock up the genome, lock down research? Science, 342: 72-73. (“Researchers say that gene patents impede data sharing and innovation; patent lawyers say there's no evidence for this”)

Jeffrey Mervis, 2013: The annual meeting: improving what isn't broken. Science, 342: 74-79. (“Annual meetings are moneymakers for most scientific societies, and scientists continue to flock to them. But as the world changes, how long can the status quo hold?”)

Diane Harley, 2013: Scholarly communication: cultural contexts, evolving models. Science, 342: 80-82.

Should there be an alternative to the Nobel Prize?

At Physics Focus, Tara Shears has a post that makes a similar suggestion to one that appeared in DTLR earlier this month, that a prize should be awarded for an accomplishment rather than a set of individuals.  Naturally my thinking is on the issue aligns well with hers.  One of the commenters to her post, John Duffield, pointed out that the Nobel committee is bound to obey the terms of Alfred Nobel's will, and that the suggestions of Shears and Sean Carroll (New York Times) should be applied to a new prize, not the Nobels.  I surely agree here too.  Thus, we'll always have the Nobels, warts and all, but their prestige will need to be rivaled by new prizes that better reflect the scientific process.  Will a benefactor step forward, wealthy but interested in reforming the reward system in science? Alternately, Science and other magazines usually publish a list of top 10 discoveries of the year or some such, and these put the focus on the achievements rather than the individuals.  Perhaps such lists, if suitably hyped up, could achieve what Carroll, Shears, and I are aiming for? 

Sabine Hossenfelder has an opposing view on the Nobel Prizes on her blog.  It consists of two lines of reasoning.  The first boils down to the following.

Giving such an honor to institutions is akin to doing away with private property in communism and believing that everybody cares for the well-being of the group as they do for their own. It doesn’t work because most people want to be recognized as individuals, not as members of collectives. That’s true also for scientists.

I found this an unexpected but quite thoughtful contribution to the conversation.  It is true that as a member of a 'collaboration', winning a prize is not exactly something one can place on one's CV.  Indeed, I found it inappropriate when individual members of the IPCC claimed to be "Nobel Laureates".  In any case, the Nobel Prizes will continue to propagate according to Alfred Nobel's will.  Perhaps having them co-exist with the Science list of top discoveries is both realistic and desirable.  The prestige and/or visibility of the Science list (or equivalent) just needs to be elevated to close to the level of the Nobel Prizes.  

I do think that this year's prize, which gave the short shrift to Kibble and robbed Brout because he had the ill fortune to die too young, still illustrate unresolved issues with the Nobel Prizes.  The Nobel committee probably would lose credibility for not recognizing in some way the discovery of the Higgs boson, but there are few good ways to do so within the constraints of Nobel's will.  Still, perhaps Kibble should have been given the third slot.  A precedent is the 2001 Nobel in Physics for Bose-Einstein condensation, which did not just go to Wieman and Cornell, who were the "first", but also to Ketterle, whose early work arguably went further than Wieman and Cornell's, but was published 4 months later.

Hossenfelder's second line of reasoning is that Nobel Laureates can be powerful spokespeople for their fields.  I don't find this one compelling.  Some of the leading spokespeople for physics are not Nobel Laureates, such as Stephen Hawking or Neil de Grasse Tyson.  These folks are much better known to the public than most living Nobel Laureates in physics.

Upcoming on DTLR

This month has by far been the busiest since DTLR began in July of this year.  And yet my posts on the three biggest fish are still in the planning stages.  So, here I just want to alert readers to the background material on two of these.

First, there is the special issue from Science on "Communicating in Science:  Pressures and Predators," published earlier this month.  Second, there is the cover story in The Economist from last week, "How Science Goes Wrong."  (See also the accompanying Leader.)  Both of these tackle central topics of this blog, and I am grateful for the high profile these have issues have taken.

The third 'big fish' of the month that I plan to comment on is a reaction to the federal government shutdown at the beginning of the month.  (Full disclosure:  I was furloughed as a result of this episode.)  I plan to muse on the role of private funding for science.

All three topics are very timely and I hope to have my comments available soon.  In the meantime, on the technology side, see this interesting post by David Auerbach at Slate on the travails of the Affordable Care Act's website rollout this month.

Software validation in computational biology

Last month in Nature, there was a brief article by Erika Check Hayden about an experiment in peer review of scientific software being carried out by the new Mozilla Science Lab. Nine papers published in PLoS Computational Biology, selected by its editors, would have their code subject to a peer review by software engineers. The experiment and its motivation are described in the article; I also recommend reading the user comments posted at the end. (See also the earlier pieces by Zeeya Merali and Nick Barnes, published together in Nature in 2010.)  Apparently there has been some controversy, as scientists are understandably nervous about having their work subjected to a new form of review. However, scientists are not well trained in software development concepts such as version control, validation, and verification, and the code they write may become difficult to maintain, or even worse, produce undetected errors that have worked their way into published research.

The Mozilla Science Lab was introduced this past summer, and is led by Kaitlin Thaney. It sponsors Greg Wilson's Software Carpentry; I strongly recommend having a look at the latter's website. I've read Wilson's essays in Computing in Science and Engineering and other publications over the years, and have been sympathetic to his views. I've heard rumors that the some of the code at Fermilab is spaghetti code, with bits and pieces of it written by many hands over many decades. Such an unwieldy mass of legacy code is almost impossible to maintain. I was told about one bug whose fix generated another, more serious bug that was impossible to debug. It was decided to restore the original bug and leave it in the code!

I am fortunate in that one of my formative experiences was an internship with a small company that, as a matter of survival, implemented a fairly disciplined software construction methodology, based in part on Steve McConnell's Code Complete. Because the company was small and had a certain rate of turnover, all of their software had to be highly maintainable, assuming the original coder was no longer employed at the firm. It was a point of pride there that you wouldn't be able to tell who wrote a piece of code found in the software they developed, without looking at the header (which had version control data), for we all conformed to the same software style.

DTLR endorses the Mozilla experiment in peer review of software. I hope we learn a lot from their experiment, even if it is deemed to be a failure in the end. In a letter to the editor, Alden and Read (2013) state that software quality should be built in from the beginning, before any data are taken, and not “inspected in” at the peer review stage. They are of course right, but to protect the rest of the community I do think software peer review is a concept that should at least be explored.

References

Nick Barnes, 2010: Publish your computer code: it is good enough. Nature, 467: 753.

Zeeya Merali, 2010: Computational science:...error. Nature, 467, 775-777.

Erika Check Hayden, 2013: Mozilla plan seeks to debug scientific code. Nature, 501: 472.

Kieran Alden and Mark Read, 2013: Scientific software needs quality control. Nature, 502: 448.

Biology's dry future

A few weeks ago, Science magazine featured a very interesting story by Robert F. Service titled “Biology's Dry Future.” The subtitle tells us, “The explosion of publicly available databases housing sequences, structures, and images allows life scientists to make fundamental discoveries without ever getting their hands 'wet' at the lab bench.” The story highlights two quotes from interviews. The first is by Atul Butte of Stanford University School of Medicine: “I'm like a kid in a candy store. There is so much we can do.” The second is by David Heckerman of Microsoft Research: “You basically don't need a wet lab to explore biology.”

The title of the story is not quite accurate. There will always need to be wet lab biologists to do experiments and generate data. What is novel here is the new breed of biologist who works on data generated by other labs, but need not have a lab themselves. This may be new to biology, but physicists have long had a split between experimentalists and theorists, recently joined by computationalists.

Of particular interest to DTLR are the three “growing pains” mentioned in the article: data access, data standardization, and genetic privacy. I will focus on the first two here. Regarding data access:

In many cases, researchers who have spent their careers generating powerful data sets are reluctant to share. They may be hoping to mine it themselves before others make discoveries based on their work. Or the data may be raw and in need of further analyses or annotation. “These are really hard problems,” Butte says. “We need better systems to reward people that share their data.”

DTLR endorses that last sentence. First of all, anyone who makes the effort to generate a good data set should make the effort to document and annotate it for use. Even if the data are never shared, pretending that it might be shared one day instills the necessary discipline for documentation and annotation. Moreover, if the work is publicly funded, then in my view the social contract requires that the data be made available to the broader scientific community at some point, perhaps after an appropriate time period of exclusive use, say, no more than two years. (This is about the time needed for a grad student or post-doc to squeeze at least one paper out of results.) The new Nature online journal for data sets would provide an excellent venue to generate a peer reviewed publication for the data set alone, rather than discoveries that can be made with it. Bear in mind that in physics, Nobel prizes are awarded to both theorists and experimentalists. Biology as a discipline should adopt a similar cultural mindset to reward both wet bench and dry bench biologists.

Regarding standardization:

Not only do research groups file their data using different software tools and file formats, but also in many cases the design of the experiments—and therefore precisely what is being measured—can differ. Butte and others argue that dealing with multiple file formats is somewhat cumbersome but that the problem is surmountable. But it can be harder to account for differences in experimental design when comparing large data sets.

DTLR could not have said it better. The core problem here is experimental design, and it will always be a limiting factor for dry lab biologists trying to combine data from more than one experiment. A similar problem exists in clinical medicine, under the term 'meta-analysis', and I'm not sure there are really good solutions there either. The best approach, in my view, is to take any findings based on multiple data sets as tentative, exploratory, and hypothesis-generating, rather than definitive. The findings should then be confirmed (or refuted) in a new experiment. This is where the dry lab biologist might have to return to the bench.

Finally, DTLR cautions that dry lab biologists should still spend some time in the lab, at least while in training. There is no substitute for bench time for getting a feel for how sloppy and imprecise experimental data can be, and where the pitfalls and potential systematic and random errors may arise from. It is too easy for a dry bench scientist to take data found in a database at face value. Spending time at the bench will provide a needed reality check.

Reference

Robert F. Service, 2013: Biology's dry future. Science, 342: 186-189.

Sexual harassment in science

Last week's issue of Nature calls for an end to sexual harassment in science. The editorial was triggered by a scandal centered on a blogger who was the editor of the Scientific American blogs. He has resigned. The editorial casts a wider net, introducing the generic character “Dr. Inappropriate” who represents “the widespread tacit acceptance of adolescent behavior.” DTLR endorses their call for fighting back against sexual harassment in science. Although I am keenly sensitive to the potential for false accusations, I condemn scientists who commit sexual harassment, particularly when they do so in the context of an unequal power relationship with the target.

As a male scientist, I have been fortunate not to have been the target of such harassment. However, as a graduate student in the late 1990s, I witnessed an incident involving two “Dr. Inappropriates” that shocked me out of my sheltered perception about the behavior of mature scientists.

The setting was as follows. Our department had a weekly seminar, and on one occasion the speaker was a distinguished physics educator from another university.  She was the author of a textbook being used in an experimental introductory class in my department. She had brought along her (female) teaching assistant (now a faculty member in her own right). Following the seminar, they invited the audience to join them in a nearby classroom where they would run a simulation of a cooperative learning class, with us as the 'students'. (Around this time I myself had already taught a couple recitation sections using the cooperative learning format.) A sales representative from the publisher of the experimental textbook was also present; she was also a female.

I was one of the more junior graduate students at the time, just getting started in research.  (At the time there were very few female faculty or students in the department; there are considerably more today.)  I sat down at one of the tables in the classroom, and was soon joined by three male physics faculty members. (Two are now retired; the third is still on the faculty as I write.) Let us call them E, K, and H.  Professor E had a strong interest in physics education, and I had worked with him in the past. The other two didn't know me very well. In any case, K took the time to introduce himself to me again, and there was some small talk. H more or less ignored me.  (I have other unflattering stories to tell about them, but those will be for another time.)

So, the simulated cooperative learning class began. We were all given an assignment to work on in our groups. Each group consisted of whoever was sitting at a table together. The seminar speaker and her assistant circulated in the room, acting as Socratic facilitators for the groups. They listened to our conversations among ourselves, and tried to help us along. E played along with the simulation. However, K and H started drifting into “adolescent behavior”. First, K commented to the rest of us that the teaching assistant, out of earshot, was cute. K and H then pretended to be 'bad students' and basically annoyed the teaching assistant when she stopped by to listen to us or help us along. Unlike E, they were not taking the exercise seriously.  

At one point, the publisher's sales representative came over, obviously interested in promoting the textbook. By now I sensed that K had lost interest in the whole exercise, but H (who was on the textbook committee) started chatting with the sales rep. I can't remember the nature of what he said, but I do remember the impression that he started getting very flirtatious with her. Her reaction was very polite and professional, and she rapidly maneuvered the conversation toward discussion of her kids. She showed H a wallet picture of the children. This put an end to the flirtation, and they started talking about the textbook instead.

The sales rep's maneuver struck me as masterful. It could not have been the first time she had to deal with this sort of thing, and she maneuvered H out of the 'adolescent' behavior firmly but without ruffling his feathers at all. I appreciate the awkward position she was in. She was obviously trying to curry favor with a member of the textbook committee, but also quick to eliminate the flirtation.

This was the first time I had seen physics professors behaving badly. I regret that I did not challenge them. It would have been risky for me to do so, but that is no excuse. A challenge is precisely what the Nature editorial recommends as a non-legalistic way to discourage such conduct. (However, based on my other interactions with K and H, I speculate they would have ignored any such challenge from me, or even attempted retaliation.) I can only imagine how much more difficult it would have been had I been a female graduate student instead. It would be totally demoralizing to witness the entire episode and understand that this is how some physics professors are capable of treating women when in a professional setting.

I ran into E a few days later, and he expressed his dismay in his colleagues. (He was also a little unhappy with me, as I pretty much said nothing throughout the whole exercise, failing to back up E in playing along with the simulation.) It is notable that even E did not feel empowered to challenge K and H in person at the time – all three were tenured. Tenured professors are untouchable.

What do I take away from this? It isn't clear whether sexual harassment by any legal definition took place. The episode I describe falls into what the Nature editorial calls a grey zone. K's comment about the 'cute' teaching assistant occurred when she was out of earshot, and didn't lead to anything further. H's flirtations ended once it was made clearly unwelcome. K's and H's 'bad student' act was not sexual harassment, but just general 'adolescent' behavior. Nonetheless, my witnessing of this episode made it possible for me to imagine that sexual harassment might indeed take place, under different circumstances, and that it could be perpetrated by people who are set up to be mentors and authority figures to me and other students. Indeed, what I observed that day seems relatively harmless compared to the Scientific American blog scandal, where a real abuse of power is alleged.

The reason the episode was a shock to me was that every single faculty member I had ever interacted with until then was, in my view, an honorable and professional person, and I had never seen professors act immaturely until that moment. Perhaps the one positive outcome was that my blissful naivete ended that afternoon.

Recommended reading

Eileen Pollack, 2013:  Why are there still so few women in science?  New York Times, Oct. 3, 2013.

UPDATE:  A bit off topic, but Sabine Hossenfelder has a (back)reaction to Pollack's piece on her blog.

October 2013 is a great month for physics fans

Of course the month began with the announcement of the Nobel Prizes in physics, chemistry, and medicine. The conversation about the physics prize in particular, discussed earlier on this blog, has been lively. Meanwhile, here in the U.S., Physics Today offered its usual bounty in this month's issue, including an article on “Measuring the Hubble Constant” by prominent astrophysicists Mario Livio and Adam Reiss, the latter a Nobel Laureate. (More on this article below.) Meanwhile, the 50^th anniversary issue of the New York Review of Books features an essay on the last 50 years or so of the development of particle physics and cosmology by Nobelist Steven Weinberg (Nov. 7, 2013 issue). Both fields, though in states of disunity and confusion in the early 1960s, gradually came to develop their own highly successful and unifying 'standard models'. The two fields also converged. Weinberg manages to tell the story without naming a single physicist, including himself.

In the U.K., it gets even better. Physics World celebrates its 25^th anniversary by publishing a spectacular special issue, which is available for free download (volume 26, number 10, October 2013). And last weekend the Financial Times' FT Weekend Magazine offered its first special issue devoted to a single science, physics (Oct. 18, 2013 issue). (Unfortunately there doesn't seem to be a single link that compiles all the online articles. Here is the lead editorial.) Finally, even The Economist covers physics in an article about the possibility of particle accelerators made of glass, which would allow them to be smaller (Oct. 19, 2013 issue).

Returning to the article by Livio and Reiss, I note with particular interest a discussion of comparing the Hubble constant H0 based on global methods, such as those made by the Planck satellite, with measurements based on local objects.

Local measurements of H0 are complementary to other, higher-redshift probes. Indeed, we'd be remiss if we did not note an apparent tension, at the 3 sigma level, between current measurements of H0 based on local objects and its deduced value based on the standard cosmological model and new Planck results for the cosmic microwave background. That tension may be the harbinger of new physics, but past experience indicates that discrepancies below 3 sigma disappear when more data are available.

Indeed, the threshold for discovery in physics is often five sigma, which was the threshold used in the discovery of the Higgs boson. As a physicist who has strayed into the life sciences, two aspects are particularly striking. First, I admire physicists' relentless skepticism of 3 sigma results (2 sigma is routinely considered 'statistically significant' in the life and behavioral sciences) and willingness to collect more data. The epidemic of non-reproducible research in the life and behavioral sciences betrays the precise opposite tendency in those fields. Second, in physics when we talk about measuring universal constants, there are often many independent procedures for measuring the same universal phenomena. This has been true throughout the history of physics. In the clinical sciences, there is usually exactly one clinically meaningful endpoint, and other substitutes (biomarkers, surrogate endpoints) may provide compelling evidence, but never sufficient in a true outcomes study. If you want to prevent a remission of cancer, you must measure the time to remission. If you want to prevent a heart attack, measure the time to the next one.

References

The Economist, 2013: Small really is beautiful. The Economist, vol. 469, no. 8858, pp. 83-84.

Mario Livio and Adam G. Reiss, 2013: Measuring the Hubble constant. Physics Today, 66 (10), 41-47.

Steven Weinberg, 2013: Physics: what we do and don't know. The New York Review of Books, vol. LX, no. 17, pp. 86-88.