How COVID-19 could make science kinder

Reviewers are providing more constructive feedback in a time of crisis, but how long will it last?

24 February 2021

VectorMine/Getty Image

In March 2020, several journal publishers introduced fast-tracked peer review for papers related to COVID-19, cutting the time from submission to publication in half.

While concerns over the move’s impact on research quality were predictable, the flow-on effect of a kinder peer review process was less so.

A new study has found that reviewers of COVID-19 papers provided more constructive feedback, such as suggestions to tone down conclusions rather than go back to the lab to do more time-consuming experiments.

“Reviewers are being more constructive and cooperative, which isn’t something that journals are demanding from them,” says lead author Serge Horbach, who studies peer review and research integrity at Aarhus University in Denmark. “It’s their individual attitudes that are changing.”

Faster but more cooperative

Horbach analyzed the review reports and editorial decision letters for 30 papers published in eLife and The BMJ. He compared reviews for COVID-19 papers published early in the pandemic, before April 2020, non-COVID-19 papers published since January 2020, and articles published one year before the pandemic.

The findings were published in Research Evaluation.

While requests for additional data or experiments were common in non-COVID and pre-pandemic reviews, they were less frequent in reviews for COVID manuscripts. COVID reviewers often instead suggested authors tone down their conclusions, clarify the methodology they used, or address limitations in the discussion section.

Even when reviewers did ask for further experiments or data from authors, they often made it only on the basis that it could be done ‘easily’ or ‘quickly’ rather than a non-negotiable requirement.

COVID reviewers also made clear suggestions for improving manuscripts, such as switching to a format that allows regular updates as new data become available. In contrast, non-COVID reviewers were more likely to offer harsher critique, along the lines of this one by a reviewer of a non-COVID article submitted to eLife : “In general, the manuscript suffers unclear and complex wording; the introduction is inflated, providing many dissociated ideas and concepts without a clear narrative.”

Horbach says the findings suggest that the role of peer reviewers has shifted to quality improvement during the pandemic rather than gatekeeping.

“It’s very important that new knowledge gets out as quickly as possible,” says Horbach. “Reviewers seem to be much more willing to assist with this, rather than just judging what’s in or out.”

A valuable lesson

Elizabeth Wolkovich, an environmental scientist at the University of British Columbia, Canada, believes the fast-tracked approach to peer review could help shift the emphasis away from publishing only novel, statistically significant findings.

“A big problem in the review and publication process is that you are encouraged to be very certain about your results so that you can get published, and yet science works on being more open about uncertainty” says Wolkovich, who discussed constructive peer review in a December 2020 blog post. “Any mechanism that breaks that down among reviewers would be really healthy for science in general.”

Taking a more constructive approach to peer review could have benefits that outlast the pandemic, says Gemma Derrick, a senior lecturer in higher education at Lancaster University, UK.

“Peer review is a quality control mechanism, but the pandemic has revealed that it’s also a very human process,” says Derrick. “This means that it can serve many more roles than just assisting decisions.”

Kinder, not easier

Derrick is currently working with the Wellcome Trust in the UK to develop clear standards for peer reviewers of early-career grant applications. Rather than simply accepting or rejecting an application, Derrick is encouraging reviewers to provide feedback that researchers can use to improve their chances in the future.

“Peer reviews at the early-career stage should be constructive and do more than provide a result,” says Derrick. “People are at an extremely precarious stage of their career, and this needs to be kept in mind when providing reviews.”

While Derrick hopes that a more constructive peer review process will become the norm post-pandemic, she says that it’s important to remember that kindness also needs to be extended to reviewers who are already time-poor.

“A kinder peer review fundamentally requires more time and effort,” says Derrick. “Being kinder is not the same as being easier.”