The debasement of the scientific literature

The debasement of the scientific literature is the collective result of many factors, such as:

• Incentives to publish in quantity rather than in quality, such as the use of impact factor to assess research merit.
• Sloppy and poorly documented methodology, leading to non-reproducible research.  Some of this is due to poor scientific education.
• Paper mills.
• Networks of authors who conspire to cite each others' works, inflating citation counts and impact factors.
• Poor editing by journal editors.
• Cursory, lazy, narrow-minded, and/or selfish peer reviewers.
• Predatory journals.
• Artificial intelligence slop.
• Scientific fraud.

Now Nature has thrown the lid open on yet another such factor, "review mills".  These are peer reviewers who produce lazy peer reviews (often reproducing identical prose across hundreds of reviewer reports) and demanding citation to their own work.  Writer Katarina Zimmer and data visualizer Andreion de Castro released this groundbreaking interactive article a few days ago to document this phenomenon, which I had not been previously aware of.  However I do not find it in the least surprising.  I won't regurgitate details here; their article is a bombshell and you should read it.

This year I was asked to peer-review an article that I had discovered had been extensively plagiarized.  The journal's editorial policy claims they check submissions against a plagiarism database, but evidently this had not been done before the manuscript was sent out for peer review.  The publisher was Springer-Nature.  Other commercial publishers (especially MDPI in a case study examined in detail) are implicated in the "review mill" problem discussed in the article by Zimmer and de Castro.  Thankfully I've never been published in an MDPI journal, as I've never had a high impression of that publisher, and I've been lucky that none of my coauthors over the decades has ever entertained publishing in their journals.  However, I have been published by Springer and the other major commercial publishers (Elsevier, Wiley, Taylor & Francis, Sage) as well as a good number of nonprofit ones.

As my scientific career may be drawing to a close, I may personally not have to deal with the debasement of the literature for much longer.  However, I remain a consumer of the literature, and find it disheartening.

A peer reviewer discovers plagiarism

In the following, certain details are anonymized to protect the confidentiality of an academic journal's peer review process.

The Story 

Last month, I was asked to serve as a peer reviewer for a manuscript submitted to a certain European academic journal, published by Springer Nature.  I have no prior history with that journal, either as an author, peer-reviewer, or even a reader.  Nonetheless, the abstract of the manuscript was sufficiently intriguing that I agreed. 

I began working on my peer review.  I found that the paper was of appallingly low quality.  I have intermittently served as a peer reviewer for various journals over the last quarter century, but I have never, until now, thought to unequivocally recommend outright rejection of a manuscript.  I began drafting extensive comments to return to the author, with constructive comments on how the paper's quality could be raised to be of publishable value.  One of the points I made was the poor engagement with prior literature, especially with the manuscript's claim that previous investigators achieved little success in predicting certain features of the world.  To challenge the author, I decided to see if I could find papers that contradicted the claim about poor prediction success.  I started to find papers that claimed good prediction success, that were not cited in the manuscript.  I then had the idea of looking up the handful of papers that were cited in the manuscript in Google Scholar, to see who else had cited them.

Lo and behold, I quickly discovered a paper published the previous October in a different journal, published by MDPI, that was nearly identical to the manuscript that I was reviewing.  A few superficial changes were made, but vast portions were word-for-word identical, including the tables and graphs.  I was shocked.  The Springer journal I was reviewing for has, on its journal policies page, a warning that all manuscripts would be checked using CrossRef's similarity checker, using iThenticate, for plagiarism.  And I confirmed that the "victim" paper was indeed in the CrossRef database, so if such a check had been run, the manuscript should have been flagged prior to it being sent out for peer review.  

I next thought about contacting the author of the "victim" paper, but I discovered an online obituary for him, indicating he had passed in December, the month before the plagiarizing manuscript was submitted to the Springer journal!  I submitted my findings to the editor, using the journal's manuscript submission portal.  About a week later I emailed him separately, as I began to worry that other peer reviewers might be wasting time on this plagiarized manuscript.

The editor replied, and indicated he would reject the manuscript.  Indeed, he did so, but in the formal decision letter (for which the reviewers were cc'd), the reports of all the reviewers were included.  Thus I was right that we had all wasted considerable time on the manuscript.  There were a total of 3 of us reviewers, and I was the only one who had discovered the plagiarism.  One of the other reviewers recommended publication, but leveled lengthy criticism on the paper; the other was much more negative, saying the paper was "potentially interesting" but leveled numerous additional criticisms.  They both complained about many of the same things I did (in my draft comments, which were not shared with the journal) though I had also identified other issues not mentioned by them (again, not shared with the journal).

After the rejection, I decided not to let my draft peer review comments go to waste, so I posted them on PubPeer as comments for the published MDPI paper (which I've been calling the "victim" paper).  Unfortunately, the author is no longer with us, and unable to reply to my criticisms.

Reflections on the Incident

Several things went wrong here.

1. The MDPI journal should not have published the original paper.  It likely would not have passed peer review at the Springer journal where it was also submitted.  Moreover, according to Wikipedia, MDPI is a controversial publisher that has been suspected of being a predatory journal.  From now on, I will refuse to submit to, or peer review for, any MDPI journal.
2. The plagiarizer should not have picked such a terrible paper to plagiarize.  Chances of getting it published would be low, except at another predatory journal.  But what was the point of plagiarizing it in the first place?  Why plagiarize?
3. I strongly suspect that the Springer journal did not follow its own policy of running a plagiarism check on the submitted manuscript.
4. The peer reviewers should and did assume that the journal would follow its policies, and none of us attempted to check the manuscript for plagiarism.  Two of the three peer reviews therefore failed to detect the plagiarism, and the third one (me) did so but not deliberately; I stumbled upon the "victim" paper while trying to accomplish an unrelated task.

Peer review is a purely voluntary effort, and journals (including for-profit journals, such as was the case here) rely on the goodwill, skill, and effort of volunteers to perform a thorough manuscript review.  When reviewers are asked to review plagiarized manuscripts, it results in a colossal waste of time and energy.  Journals often have trouble convincing researchers to agree to review manuscripts, and when incidents like this occur, it decreases the reputation of the journal and creates resentment on the part of reviewers whose time was wasted.

Thoughts on graduate education in physics

The American Journal of Physics this month features an article by Laurie McNeil, who was given a teaching award named after J. D. Jackson.  The article is based on her acceptance speech for the award, in which she mentions never having taken a full grad course based on Jackson's classic (and notorious) electromagnetism textbook.  But the important point about her article is that it offers a perspective on graduate physics education that focuses not on reproducing physics professors, but on providing success skills for its graduates, most of whom do not make permanent careers in academia.  She does something that few physics departments bother to do - she tried to track down the current positions of all her department's alumni since 1981.  Most physicists only keep track of their graduates who go on to research/academic careers, as she notes, and deem the rest "lost to the profession".

I fully agree with the article.  I should note, though, that the article is a high-level, vision type piece, without a lot of nuts and bolts.  For example, I would advocate that graduate education in physics should include an allowance, if not strong encouragement, for students to take a break to do a summer internship or two in industry, government, or the nonprofit sector.  This is something I've previously discussed on this blog.  If I sat down and thought hard about it, I would add additional recommendations.  The currently common form of physics graduate education is in my view ritualistic and inadequate for purpose.

 

 

More evidence of the pollution of the scientific literature

DTLR has often lamented the pollution of the scientific literature by papers with flawed methodology or reporting, including bad study designs (e.g., lack of randomization in preclinical comparative studies) or bad analyses.  The inclusion of such published but flawed research has an amplifying effect when it is included in meta-analyses and systematic reviews.

Now comes a pair of news articles, one in Science by Holly Else, and the latest from Nature by Richard van Noorden, reporting that the inclusion of fake papers has even further polluted the literature and undermined such systematic reviews and meta-analyses.  This is an indictment of the existing mechanisms to fund, peer-review, and publish scientific research. 

The DTLR blog was founded partly to provide discussion of the eroded quality of the scientific literature, but it seems in the decade since its existence, the problem has only gotten worse.  And the public cannot be asked to "trust" science until scientists themselves can trust what their colleagues are doing.

What is the point of trying to do scientific research, when even the most rigorous and careful scientists will see their work published alongside flawed or even completely faked "research"?  Why are society's resources being funneled into an enterprise that is increasingly becoming a charade instead of a genuine contribution to human knowledge?

This is a very discouraging way to bring 2024 to an end!

Peer Review Week 2024

Apparently we are ending this year's "Peer Review Week".  Many publishers are celebrating the work of their peer reviewers, who are typically unpaid volunteers that support the review of articles submitted to academic journals, grants proposals submitted to funding agencies, and occasionally papers submitted for presentation at conferences.

For those of us who work in non-academic institutions, like private sector or government organizations, there is often a parallel internal peer review process for employees, before they are allowed to submit to an external journal or conference.  There are notable differences between this internal peer review process and the academic reviews I refer to above; the rest of this post will focus solely on the academic sector review.

Peer review has taken a great deal of criticism in recent years.  I personally am sympathetic to the view of Adam Mastroianni, who argues that no peer review is better than (at least conventional) peer review.  Post-publication peer review (such as on PubPeer) is, for me, a preferred way for a community to assess the merits of a scientific publication.  Peer review, as conventionally practiced, does not do what it claims to do, confer an imprimatur of quality and approval upon a scientific publication.  There is no limit to the stories of terrible papers published in top journals that contain almost blindingly obvious flaws, along with excellent papers that were repeatedly rejected by outstanding journals.  A defender of peer review may argue that surely the false positive and false negative error rates must be low.  It is impossible to verify such a claim because the peer review process is shrounded in anonymity, and publishers are reluctant to release any data regarding the performance of peer review at their journals.

I don't see anything of value that pre-publication peer review actually create that cannot be duplicated, in a better way, using post-publication peer review.

A thought-provoking podcast given by Dr. Mastroianni can be found at Econtalk (along with a reference to his substack article on the topic).  I won't rehearse all his criticisms here - he does it more eloquently than I could.

One issue not addressed by him is the inclusion of ad hominem attacks on the authors of manuscripts in reviewer reports.  The U.K.'s Institute of Physics Publishing (IOPP) released last week a campaign against such unprofessional comments in peer review.  This podcast is an extremely interesting discussion of the topic, and the proactive ways IOPP is using to address it.  IOPP also rates each review they receive, and is now willing to share this feedback with reviewers who request it.  It's about time someone reviewed the reviewers!  (I may well decide to take the peer review training course offered by IOPP if time allows.)

I've been on both sides of peer review on many occasions.  I've had experiences where the peer review process clearly improved my manuscript, and I've had experiences where the process clearly degraded the manuscript, that is, the published version is notably worse than the original, in my not-so-humble opinion.  I've experienced a few cases where reviewers of my work have signed their reviews, i.e., revealed their identities; most of the time they are anonymous, but in a handful of cases, I believed I could guess who some of them are.  

I've also heard from others who've had their ideas stolen by peer reviewers!

On the other hand, in my highly biased opinion, every peer review I have performed for others has resulted in improvements in their manuscripts.  It is notable that every article I have ever peer reviewed has eventually been accepted for publication.  On the other hand, I've never signed a review, so I've always been anonymous to the authors I've reviewed.

Finally, in a handful of cases I've posted non-anonymous post-publicaton peer reviews on PubPeer. And sometimes when I wonder about the legitimacy of a published paper, I'll check up on it in PubPeer.  I advise readers to do the same.

Hacking Google Scholar

Finally, we enjoyed (but were dismayed by) the account given on Reese Richardson's blog about how Google Scholar can be "hacked".  Phony papers are uploaded onto ResearchGate, to boost citations to one's own (possibly equally fraudulent) papers.  There are companies offering to carry out such a citation-boosting campaign for paying customers, but Richardson shows how easily one can do it oneself.  However after he wrote about it, Google Scholar removed the fraudulent citations to the work of "Larry Richardson" (a cat), without necessarily implementing a global solution to the problem.

Predatory conferences: an expose!

Also in this week's Nature, an investigation of predatory conferences!  Journalist Christine Ro actually attended one.  Her observations are eye opening.

I get spam emails from these conferences on a weekly basis.  I've never been tempted to attend one.  I tend to go to conferences sponsored by professional societies that I know fairly well, or other established conference series.  However, as the article shows, many do not have the luxury of being as picky as I am.  Predatory conferences are basically rip-offs, of both the attendees and those who evaluate their CVs.

DTLR hereby denounces predatory conferences, and urges scientists to boycott these sham events.

Happy Birthday, APS!

May 20 was the 125th birthday of the American Physical Society (APS).  Founded in 1899 at Columbia University, and later taking over publication of the Physical Review (founded in 1893 at Cornell University) in 1913, the society has expanded both its core early functions (conferences and journal publications) as well as added many other programs in the last century and a quarter.

I joined the APS around 1997, and its Centennial Meeting in Atlanta (1999) was the first national scientific conference I ever attended, and the first one I ever gave a talk at.  That year, the March and April meetings were combined into a grand extravaganza, with (if I recall right) well over 10,000 attendees.  I remember following (by coincidence) at least one Nobel Laureate down an escalator (J. Friedman) and recognizing others in the hallways or meeting rooms.  I presented again at the 2002 March Meeting in Indianapolis, though my findings were shot down by a very prominent scientist in the audience.  That was the second and final time I presented at an APS conference, though in later years I've attended "for fun" on my own time on several occasions, including last year.  I have also attended a number of APS section meetings, as well as the Division of Fluid Dynamics annual meetings.  I once published in an APS journal (Physical Review E), but that was during the end of my time in graduate school.

Since I have not been a practicing physicist for over two decades, I have not been an active APS member, but I am a Life Member of the parent organization and a number of its individual units.  Indeed, I have been an APS member for just over 20% of APS's own lifetime, and just over 50% of my own lifetime!  Many Happy Returns, APS!

Another howler from Science Magazine

In my last post, I scolded the editor-in-chief of Science, Holden Thorp, for an op-ed, though I largely agreed with his conclusion.  It didn't take long for him to provoke the DTLR hornet's nest again, and this time I can't say I agree with him.  The April 11 issue features another op-ed by him, titled "Teach Philosophy of Science".  He points out the risk of the erosion of public trust in science, pointing specifically to survey results indicating that 92% of Americans "felt it important that scientists show they are 'open to changing their minds based on new evidence,' which is of course what they must do," Thorp writes.  He claims that the "history of science is a powerful narrative of this culture of self-correction" and laments that "Resetting the public's understanding of how science works will be a big job."  His proposed solution is to teach philosophy of science to undergraduate science majors!

I regard that proposed solution as a non-sequitur.  Using Thorp's own argument, I say it is the history, not the philosophy of science that should be taught.  To be honest though, when I was a young student, I was interested in history, but not the history of science.  I was busy learning the science itself.  I am much more receptive to history of science now, for the reasons Thorp suggests.  I have not found philosophy of science to be insightful or to help me think about doing science.  Earlier this year I finished reading A Philospher Looks at Science, by Nancy Cartwright.  While I admire her support for the views of Kay and King's Radical Uncertainty, her book hardly made a dent on my way of thinking about science, and how to do science, whereas Kay and King helped me substantially clarify my thinking on probability models applied to the real world.

Returning to Thorp, the bigger problem in his argument is that somehow the public will trust science more if only they understood better how scientists change their views and "self-correct".  Unfortunately the history of science has plenty of examples of abject failures as well as successes.  Consider the wholly unnecesssary resistance to such figures as Ignaz Semmelweiss (who contributed to the germ theory of disease), Dan Shechtman (discoverer of quasicrystals), and Katalin Kariko (mRNA pioneer).  Consider also the crisis of non-reproducible research that I mentioned in my last post.  Science won't self-correct if no one bothers doing the self-correcting.  There are no financial or professional incentives either to reproduce others' work, nor to do one's own work with sufficient care and sound methodology to ensure it will withstand the test of time.

At its most cynical, science is merely a self-propagating exercise in securing endless piles of grant money for "research" of limited value and interest, as noted in this recent video by Sabine Hossenfelder, who points out that it is public tax money that fuels the seemingly joyless cycle of beg, spend, and publish.  If the public saw science this way, rather than in the romanticized way Thorp imagines, would they really trust scientists enough to allow their taxes to be spent as such?  Personally I would not, and I am a scientist!

I assert that the number one thing scientists can do to increase public trust in science is to reform their own house.  Only then will they deserve public support and trust.

Digression.  I last wrote about Sabine Hossenfelder nine years ago this month.  Although in that post I largely disagreed with her on one particular topic, I've since come to admire her devastingly blunt contrarian views on physics research.  For some time, I was also a financial supporter of hers on Patreon, though I am not longer financially able to support any of the deserving people I admire on Patreon.

Science Magazine colossal hypocrisy

I could not believe my eyes when I opened the Feb. 2 issue of Science to the editorial page.  The featured op-ed, as frequently is the case, is by the current editor-in-chief, Holden Thorp, and is entitled "Earning Respect and Trust".  He discussed the Science family of journals' editorial staffs, and the lack of respect they sometimes encounter by authors of submitted manuscripts.  He rightfully defends these staffers, and insists that they be respected by lab scientists.  I wholly agree with these sentiments.  It is his method of argument that I strongly reject.

Let's look, for example, at a few sentences in the third paragraph, line by line.  First,

If anything, the challenges that science is experiencing now are not due to a lack of success in the laboratory.

That is manifestly false.  The crisis of non-reproducible research, documented for nearly 20 years now, with almost no widespread progress on reform, has been a frequent topic of discussion on DTLR, and even in the news section of Science itself.  Two of the cornerstone enablers of this crisis have been poor methodology, both in the lab and in the data analysis.  I have specifically called out previous AAAS CEO Rush Holt's similar denials that it is exactly the core activity of scientific research that is rotten; for example:  here, here, here, and here.  I won't rehash those arguments.  Holt's retirement from AAAS, the year after my posts, was greatly welcomed.  However. Holden Thorp has now revived what I will call the Holt Myth.

Next,

The notion that lab work is the only purposeful endeavor in science is obtuse and is an example of precisely what leads to the view that scientists are intellectual elites who do not value the contributions or abilities of anyone except themselves and the small group they deign to recognize as their peers.  Every time this academic hauteur is revealed to the public, confidence is lost for a simple reason--scientists like these are not inspiring the people's trust.

Now, I completely agree with this criticism!  However, he uses the term "top journal" earlier in the paragraph, not quite saying, but seemingly implying, that his journal is among the top ones.  In many fields Science certainly is considered among the top journals (e.g., see here), and he is part of the very elite he seems to be complaining about.  Authors who aim to publish in such elite journals, where novelty is prized over reproducibility, are incentivized to be sloppy and fast, rather than careful and slow.  The motto of DTLR is "Garbage In, Gospel Out":  Thorp's journal could be exhibit A of this phenomenon.  Perhaps Science and its editors should themselves take a humility pill before lecturing the rest of us on the harms of elitism.

 

 

Peer Review

Last week in the online magazine Physics, psychologist Simine Vizire published a call for peer review reform.  Hers is not the first, nor will it be the last.  My personal experience with peer review is mixed and I would support calls for reform.  In discussions with other scientists, I have heard appalling examples of abuse, such as the stealing of authors' work and credit by anonymous reviewers.

All the manuscripts I have submitted for peer review have benefitted from the process.  The published manuscripts are usually improved as a result of the vetting, and in most cases I prefer the published version over the initially submitted draft.  However, in two cases the published version is also shorter or more condensed than I would have preferred, and 19 and 12 years later, respectively, I'm still sore about that.  In one case, the peer review process extended for over a year, in my opinion needlessly, due to a single obstinate reviewer who added absolutely no value to the process.  In another case I eventually published a manuscript in unrefereed form (in a conference proceedings) because the journal rejected it for being "too simple".  (The talk corresponding to that paper won an award at the conference where I presented it!)  I have since pulled the same trick for a second paper, also rejected, though it gets lots of reads where I posted it (academia.edu).  I strongly believe that the taste of referees and editors does not necessarily correspond with that of readers.

As a referee, I also feel that I strongly improved every manuscript I have refereed.  However, I have seen other published research that should have been better vetted.  I have taken to posting critiques of such papers on PubPeer, a website for open, post-publication "peer review".  In my view, tools like PubPeer should be better integrated into the research culture.  Upon reading a paper that you think you may want to use, one of your first instincts should be to look it up in PubPeer.  And scientists should not hesitate to post their criticisms or discussion of published work on PubPeer.

Vazire makes some pertinent points.  We don't understand the performance of peer review because there is essentially no empirical research done on this (with the exception, she says, of medical journals) - a hypocrisy considering scientists pride themselves on being "evidence based".  She calls for giving peer review researchers access to now-confidential materials, the raw material of peer review, so as to better study the process.  (It would be a brave editor who agrees to such sharing!)

Other issues she complains about (status bias and bias due to the gender, institution, or ethnicity of authors) could be addressed by double blind peer review, a process I have participated in myself.  She calls for more diversity in recruiting referees, though the flip side of this is placing additional (unpaid) labor burden on minorities, who already bear an additional burden at their institutions in many cases.  I agree with her that mechanisms to hold editors accountable for their own abuses are nowhere near sufficient.  

I support Vazire's recommendation of disclosure of referee reports and decision letters; but such documents are taken out of context without the corresponding draft versions of manuscripts.  Thus the use of preprint servers and tools like GitHub could be used in conjunction with transparency of peer review.

Vazire also requests specialized review processes; statistical analysis is her example.  In principle this is a sound suggestion, but in practice I frankly believe statisticians might do both harm and good in this role.  Much of the damage to the research enterprise in the first place has been due to the influence of the statistical inferential framework, as I've written about previously.

Peer review is a broken process, but it should be reformed instead of discarded.  Statistical inference too is broken, but we've given statisticians a chance to reform it, to no avail.  Discarding it in most cases would be a step forward.

Excitement, self-deception, and retraction

Yesterday, Davide Castelvecchi reported in Nature on the retraction of a 2018 paper claiming detection of a signature suggesting the existence of a Majorana fermion state.  It's a good article, and includes a link to the report of an independent investigation which concluded that, while no fraud had occurred, the authors of the paper essentially fooled themselves, implying that they fooled everyone else too, by publishing.  There are a couple excerpts from the report (by "experts" P. Brouwer, K. Ensslin, D. Goldhaber-Gordon, and P. Lee) that are worthy of reading here.

First, from Section 3:

And finally, from the Conclusion:

It should be noted that the expert investigators were provided with unpublished data, available to the authors before they published, that would have cast doubt on the original authors' conclusions, and that even the published data had features that a sharp reader could have used to cast such doubt.

Matt Ridley on "What the Pandemic has taught us about Science"

This weekend's issue of the Wall Street Journal features an insightful essay by Matt Ridley, in the Review section, titled "What the Pandemic has taught us about Science".  Two of the best passages are the following very simple observations:

"Seeing science as a game of guess-and-test clarifies what has been happening these past months.  Science is not about pronouncing with certainty on the known facts of the world; it is about exploring the unknown by testing guesses, some of which prove wrong."

"The health of science depends on tolerating, even encouraging, at least some disagreement.  In practice, science is prevented from turning into religion not by asking scientists to challenge their own theories but by getting them to challenge each other, sometimes with gusto."

I strongly recommend the article to DTLR readers, even if you may nit-pick a few details.

Escape From Model-Land

Earlier this week, Mark Buchanan's column in Nature Physics featured "The Limits of a Model" , concerning  the pitfalls of epidemiological forecasting, of course a topic very much in the news in the last few months.  Near the end Buchanan cites an intriguing paper by Thompson & Smith (2019), "Escape From Model-Land", a bracing discussion of why one should be very humble about using mathematical/empirical models.  It calls to mind another entertaining piece from over a decade ago, the "Financial Modeler's Manifesto" by Emanuel Derman and Paul Wilmott.

DTLR strongly recommends the Thompson & Smith piece, a candidate for required reading by every applied mathematician, applied statistician, and mathematical/computational modeler in every discipline.

Reference

Erica L. Thompson and Leonard A. Smith, 2019:  Escape From Model-Land.  Economics:  The Open-Access, Open-Assessment E-Journal, 13:  2019-40.

In praise of high journal acceptance rates?

A Penn State professor of astronomy and astrophysics, Jason Wright, published a commentary in the February 2020 issue of Physics Today, titled "High journal acceptance rates are good for science".  He notes that the journals he publishes in have an 85% acceptance rate.  He says that "Therefore nearly all significant astronomical results submitted to those journals that are not obviously fatally flawed are likely to be published....it is the sign of a healthy culture of science, and astronomy is better for it."  He writes about how this state of affairs helps to avoid publication bias, because paper rejection is often influenced by reasons other than quality:  "scientific taste, politics, professional advantage, and science's inherent conservatism."  He even writes that "it's important that scientists be allowed to be wrong in the literature, as long as they have made no errors."  He concludes that "referees best serve when they act not as gatekeepers but as editorial consultants and independent voices that offer construcive criticism that improves submitted papers."

DTLR finds much merit in Wright's arguments.  However, I do think referees and editors have to be gatekeepers when it comes to one point that Wright alludes to, but does not emphasize.  I and many others have seen methodologically unsound research published, even in prestigious journals with low acceptance rates, in the life and social sciences.  A methdological critic would have been able to reject the paper before seeing any of the data.  This is a collective failing of authors, referees, and editors, who are often themselves untutored in even basic principles of research design and execution. There has been much discussion of these phenomena, for instance, in a special issue of the Lancet in 2014, and in closely related discussions of reproducible research (for instance).  Sadly, that this has continued to be a recognized problem for almost 15 years is a sign of an unhealthy culture of science.

One proposed remedy has much appeal:  "Results-Blind Manuscript Evaluation" (RBME), initially proposed by Joseph Locascio over 20 years ago (eg, Locascio, 2019).  This involves a two-stage manuscript review, where a mansucript is first evaluated on the basis of the Introduction and Methods sections, without knowledge of the results.  Methodologically unsound research can be rejected out of hand at this stage; if not, the manuscript moves to the second stage where its entirety is reviewed, though the decision to accept or reject may still not be based on the results, but only on the soundness of execution, analysis, and presentation thereof.  See the cited paper by Locascio for details.  RBME would not solve all the problems, but would go a long way in changing the incentives.

DTLR believes that a scientific journal should combine the insights of Wright and Locascio in its efforts to fight publication bias, while ensuring that research with a chance of contributing to, rather than misleading, the work of others, sees the light of day.

References

J. J. Locascio, 2019:  The impact of results blind science publishing on statistical consultation and collaboration.  The American Statistician, 73 sup 1:  346-351.

J. Wright, 2020:  High journal acceptance rates are good for science.  Physics Today, 73 (2):  10-11.

The Women of Fluid Mechanics: Personal Stories and Practical Advice

Last month's issue of APS News featured an article by Otani & Hu, titled "The Women of Fluid Mechanics:  Personal Stories and Practical Advice".  The article was a commentary based on a panel session at last year's annual APS Division of Fluid Dynamics (APS-DFD) meeting, which featured four female professionals in the field (3 faculty members, and the fourth the founder of a very popular blog and miniature social media empire, FYFD).  (Disclosure:  I am a "patron" of FYFD through the Patreon website.)

Among other things, the authors write that "In spite of their intellect, work ethic, and accomplishments, all of these women have faced and witnessed disrespect and scrutiny based on their gender."  It is indeed disappointing that this still happens at the end of the second decade of the 21st century.   I myself witnessed inappropriate behavior by male physics faculty in the late 1990s, as I wrote about here, though that episode was not related to fluid mechanics. 

DTLR supports efforts to reduce the barriers for women in fluid mechanics research.  One female fluid dynamicst (whom I have never met) certainly has had a key influence, through her published work, on my brief years of research in fluid mechanics.  She is cited in every fluid dynamics paper I have co-authored, and her name appears in the abstract of 3 of the 4 refereed papers of mine in the field.  Hopefully more women's names will appear in the reference lists of fluid dynamics research papers as time goes on.

Reference

Courtney Otani and David Hu, 2020:  The women of fluid mechanics:  personal stories and practical advice.  APS News, 29 (3):  8.

Scientific American on "How to Fix Science"

The October 2018 issue of Scientific American features a set of articles under the umbrella, "How to Fix Science."  It begins with a prologue describing "antiscience currents" and arguing that one of the ways to "fight back" is to "tackle its own problems" and "shore up their enterprise from the inside."  The articles in the feature are as follows.

• "Rethink funding" by John. P. A. Ioannidis.
• "Make research reproducible" by Shannon Palus.
• "End harassment" by Clara Moskowitz.
• "Help young scientists" by Rebecca Boyle.
• "Break down silos" by Graham A. J. Worthy and Cherie L. Yestrebsky.

The middle three articles are by science journalists, while the first and last are by academic scientists. Readers of DTLR will recognize that the first two articles discuss topics that have been kicked around on this blog over the years.  DTLR readers will also notice that I have been castigating Dr. Rush Holt, CEO of AAAS, recently, for avoiding these very issues.  Holt believes science is just fine the way it is.  DTLR has previously made a similar argument as the prologue of the Scientific American feature:  we need to clean up our own house before we can, with any good conscience, ask the public to continue financially supporting our endeavors.  Thus, I view the Scientific American piece as a rebuke to the Rush Holt model for responding to the "significant erosion of trust in science over recent years" (a phrase used by Worthy & Yestrebsky).

The leading piece by Ioannidis is a good starting point for this conversation.  Perverse incentives stimulate perverse outcomes.  His piece not only touches on funding issues, but hiring, promotion, and tenure decisions. While I disagree with many of the details in his proposed solutions, I believe the spirit of his piece is absolutely on target.  We need to fix the incentive systems in the scientific enterprise, and this is fundamental infrastructure.  Unfortunately the elites who thrive in the current scientific infrastructure have a vested interest in maintaining the status quo, and would oppose all Ioannidis' proposed reforms.


The article by Palus is a decent introduction to the non-reproducibility issue for the general reader, but lacks the space to delve into the methodological details of good experimental design, experimental quality control, and adequate reporting that are discussed, for instance, in Richard Harris' recent book, Rigor Mortis.

Moskowitz states that sexual harassment is more prevalent in academia than in any other arena except the military.  I am not surprised by this.  Having worked in various sectors of the economy - academia, industry, and government - my personal preference for a setting conducive to scientific research is the private sector, not academia.  In fact, the academic tenure system protects 'assholes' including those who practice harassment.

Boyle's article consists of comments by various young scientists on the following issues:  moving, money, culture, family, industry vs. academia, getting jobs/fellowships/into school, and representation and inequality.  I agree with the writer as well as Ioannidis that the treatment of young scientists in today's science ecosystem is appalling, and needs to be drastically reformed.

Finally, the article on interdisciplinary research is really another reflection of the perverse incentive systems for academic science.

I mentioned that I agreed with the spirit, but not many of the details of Ioannidis' piece.  Readers, what problems do you see in the incentive structure for scientists, and what solutions would you propose?

More pointless AAAS propagandizing

In the last few months, I have written about Dr. Rush Holt's stewardship as CEO of the American Association for the Advancement of Science (AAAS).  Today I received a fund raising letter, mimicking the language of his earlier writings, but giving examples of how AAAS would "promote and support good science communication."    All of this again presumes that the "scattered distrust of science" is due to poor communication and public misunderstanding.  That may well be, but first and foremost Science has to clean up its own house.  The current incentive system for research penalizes the care required to increase its reproducibility.  Using the considerable influence of the AAAS to help address this does not seem to be a priority of Dr. Holt, who insists that science is just fine the way it is.  If you received a similar fundraising letter from Dr. Holt, DTLR encourages you to send it to the shredder.

Physician, Heal Thyself

Just today, Dr. Rush Holt (CEO of AAAS) sent out the following message.


DTLR does not recommend responding to this solicitation.  Until AAAS is prepared to dedicate an equal amount of resources to cleaning up Science's own house - by working to reform the perverse incentive system for scientific research, and improve its reproducibility - the above solicitation is nothing but a self-serving propaganda effort.  AAAS, as publisher of Science, the nation's leading general science journal, has a particular responsibility here, and Holt claims to be its Executive Publisher.  Will Science proceed to make the distinction between exploratory and confirmatory research, as its previous Editor envisioned?  In the midst of the crisis on reproducible research, discussed at length on this blog, how much longer will Science deserve the public's trust and support?

See also this post by Sabine Hossenfelder.  I have disagreed with her in the past, but this time I'm 100% on board.