Q&A: Potential bias in scientific publication
IIn reality, scientific publication is an objective enterprise in which manuscripts are subjected to high standards of review to ensure accuracy and guard against conflicts of interest that could compromise the reliability of a study. Yet as Retraction watch and sometimes other press materials, it is not uncommon for poor quality, or sometimes fraudulent or senseless articles to obtain the imprimatur of publication in a peer-reviewed journal.
See “The best retractions of 2020”
A study published today (23 November) in PLOS Biology highlights the potential favoritism that could be present in the editorial procedures of specific journals, allowing less-than-stellar articles to pass. Clinical pharmacologist Clara Locher and a team of researchers from the University of Rennes in France examined nearly 5 million articles published between 2015 and 2019 in 5,468 journals and discovered that while the majority of journals published publications distributed between a large number of authors, five percent of the journals had a single very prolific author who was responsible for at least 11 percent of the articles published in the journal. Additionally, in a random sample of this subset of journals, the highly published author was on the editorial board 61 percent of the time, and their papers were accepted within a median of three weeks after submission, a rate much faster than the average of the Over 100 days reported in a Nature article on studies published in Nature and PLOS ONE (no paper quality assessment was performed). The authors of the new survey argue in their article that greater transparency is needed around the editorial practices of journals.
See “Revealing Peer Review Identities Could Introduce Bias: Study”
In an email interview with The scientist, Locher discussed his investigation and what the results suggest about some biomedical research journals.
The scientist: What inspired you and your team to investigate favoritism in research publications?
Clara locher: We began to explore favoritism in research publications following the hydroxychloroquine saga. . . . A common thread among the early articles supporting the use of hydroxychloroquine in COVID-19 was that these articles were all published in journals where at least one author was on the editorial board or, indeed, editors. In addition, the time between submission and acceptance was unusually short as articles fell below general research standards. All in all, these elements cast doubt on the quality of the editorial process.
See “Journal Editor Concerned About Hydroxychloroquine Study”
ST: Can you explain to me your process for evaluating favoritism in publications? How did you decide what to assess regarding the paternity data?
CL: Among the articles supporting the use of hydroxychloroquine, a so-called meta-analysis has been published in New germs and new infections while the scope of this review does not really correspond to the therapeutic issues. So we took a closer look at this review and found that 35% of all articles were published by at least one author on the current Editorial Board. A value that is not expected!
Looking at other authors of the meta-analysis, we found that Didier Raoult signed [authored] 235 of 728 articles published in New germs and new infections, making him the most prolific author of this journal. Didier Raoult is not part of the editorial board, but as director of the Institut Hospitalo-Universitaire Méditerranée Infection, six members of the editorial board ([including] the editor-in-chief and the deputy editor-in-chief) report to him. Coincidentally, Dorothy Bishop reported a similar analysis in her blog on the field of psychology. These convergent analyzes have led us to consider the “percentage of articles by the most prolific author” as a potential warning to identify journals suspected of questionable editorial practices.
ST: Based on this, what is the Gini index and what does it tell you about authorship in journals?
CL: The first caveat we identified, the “percentage of articles by the most prolific author”, focuses on a single author and is sensitive to the number of annual publications of the journal. This is why Alexandre Scanff proposed to supplement it with the Gini index, a statistical measure widely used in econometrics to [evaluate] the level of inequality in the distribution of income. In our study, “income” corresponds to the number of articles signed by the authors in a given journal. The advantage of this measure is that it makes it possible to identify journals for which a group of authors monopolizes the authorship.
ST: Did any of your results surprise you?
CL: Yes and no! The impression is that we can have cases, for different reasons, of favoritism in editorial decision-making, and this is not new. What is new and surprising is that in a subset of journals, indexed in the NLM [National Library of Medicine] catalog, a few authors, often members of the editorial board, are responsible for a disproportionate number of publications.
ST: What are the limits of this study?
CL: The main limitation of our study is that these quantitative metrics are not sufficient to claim that there is biased editorial decision making. These metrics should be seen as warnings that should lead to a more detailed analysis of the journal: qualitative analysis of the articles published in this journal, and inspection of the place of prolific authors in the editorial board. This careful inspection of the journal should eliminate false positives represented by active editors and / or professional journalists.
On the flip side, these quantitative metrics might only point to the tip of the iceberg by identifying only extreme cases. This is particularly the case with the “percentage of articles by the most prolific author”: the more the number of articles published by a given journal increases, the more difficult it is for an author to sign 10% or more.
ST: Why do editorial biases and potential nepotism in research journals harm the research community, and in particular the biomedical research community?
CL: As long as researchers are rewarded based on measures of productivity, favoritism in journal editorial procedures could be considered unethical. In fact, these reviews could be used to increase productivity-based measures such as number of publications, number of citations, with a positive effect on decisions about promotion, tenure and funding of grants. In addition, members of the editorial board can use their position to publish articles that do not meet the standard of quality required for publication. In the case of this biomedical research, it can have negative consequences for evidence-based medicine, as we have all witnessed in the case of hydroxychloroquine and COVID-19.
ST: What are the future steps you want to take from this research?
CL: Our investigation provides information on the broad scene of what we call “nepotist newspapers”. The next step is therefore to describe this phenomenon [in a fine-grained way], in particular by studying the quality and the integrity of the publications of the editors in their own journal.
ST: What do you hope readers, research publications and their editorial boards to take away from your findings?
CL: We hope that both readers and editorial boards will realize the need to build confidence in editorial practices. For this reason, journals should be transparent about their editorial and peer review practices.
Editor’s Note: This interview has been edited for brevity.