Should scientists handle retractions differently?

Peer review cannot catch everything. In many papers, there is no peer review at all, it is editorial review that checks off a few boxes and relies on post-publication peer review to find flaws. That makes retractions more common.

But the term itself, and the meaning, may be overblown. Just like in BPA or pesticide studies, exaggerated claims about retractions, legitimate or not, can lead to policy actions that hurt science and public acceptance of science. In 2005, for example, South Korean researcher Woo-Suk Hwang and colleagues made international news by claiming that they had produced embryonic stem cells from a cloned human embryo using nuclear transfer - science that the Obama administration still bans in the United States. Within a year, the work had been debunked and was followed by findings of fraud. South Korea put a moratorium on stem-cell research funding and so scientists abandoned or reduced their work in the field. 

Yet by 2007, other stem-cell researchers had found that the debunked research contained a few solid findings amid the false claims. Nonethekess, it took time to rebuild support for the field. 

MIT scholars have quantified the fallout for scientists whose fields suffer high-profile retractions, with a twist: Even valid older research, when cited in a retracted study, loses credibility — especially if the retracted paper involves malfeasance. The fallout from a retraction does not land solely on the scientists who are at fault, but on people in the field more broadly. Because DDT was once believed to be harmful, all pesticides get a critical eye. Today, the EPA shows countries with malaria how to spray DDT inside their homes, but it remains banned domestically and DDT remains one of the few pesticides everyone in the public can name.

The new paper contends, "scientific misconduct and mistakes, as signaled to the scientific community through retractions, cause a relative decline in the vitality of neighboring intellectual fields." This spillover effect, which includes a 6 percent decline in citations relative to similar, unaffected papers, suggests that scientists would benefit by trying to describe the nature of each retraction in more detail.

"A well-functioning, transparent retraction process is actually part and parcel of the scientific system," says Pierre Azoulay, an economist at the MIT Sloan School of Management, and a co-author of the new study. "We need a system where … journals help the readers spell out the reasons for the retractions, and help the scientific community parse the implications for the forward movement of science."

Identifying the "stigma story"

The paper, "Retractions," is published in the Review of Economics and Statistics, a peer-reviewed economics journal. The authors are Azoulay, the Sloan Distinguished Associate Professor of Management; Fiona Murray, the Alvin J. Siteman Professor of Entrepreneurship, associate dean for innovation at MIT Sloan, and co-director of MIT's Innovation Initiative; Joshua Krieger, a doctoral student at MIT Sloan; and Jeffrey Furman, an economist at Boston University.

Murray and Furman were co-authors of an influential 2012 paper that studied the circumstances in which retractions occur. The new paper, Azoulay notes, is different in that it spotlights "the consequences of retractions, not their antecedents."

The current study focuses on the life sciences. The researchers examined the effects of more than 1,000 retractions of papers published between 1973 and 2007, and retracted by 2009. They also used the PubMed Related Citations Algorithm to help define the research fields relevant to those retracted papers.

Given those definitions of research fields, the researchers then examined the effect of retractions on intellectually related work. One group of 60,000 papers that they examined experienced the 6 percent decline in citation rates after retractions occurred in the same fields. To establish that fact, the researchers compared the citation trajectories of those papers with a control group of 110,000 papers that were published in the same journals. The data and method draw on work Azoulay has compiled and refined while developing numerous other analyses of citation rates in scientific literature.

But within these numbers lie another story: The citation rates for related papers that are still valid drop more precipitously when papers citing them are retracted for reasons of fraud or other misconduct, as opposed to, say, a laboratory mistake.

"Most of the declines in citations and funding we see are driven by fraud cases," Krieger notes. "When an honest mistake happens, the related field doesn't experience this big decline."

Conversely, this means that, given two equally valid sets of past research, legitimate research cited in a paper later retracted for fraud will be more harshly punished.

Further evidence for the "stigma story," as Krieger calls it, is that academic papers and nonacademic papers from firms tend to shun retracted papers at similar rates — but papers from commercial firms do not avoid citing older, still-valid related papers to the same extent as academic researchers. This could mean there is an unfounded flight away from those related papers in academic research.

"Our evidence pointing to the stigma story implies that funding agencies and investigators should be more cautious when deciding to abandon a field after a case of research misconduct," Krieger adds. "We need to be careful in separating what we've learned about the field's scientific status from our strong reactions to the disgrace of misconduct and fraud."

Limits to knowledge

As Azoulay and his colleagues acknowledge, there are limits to their study. It's possible that citations in fields affected by fraud logically should decline more than those marred by honest mistakes, because scientists may rightfully conclude that research areas containing outright fraud are further away from yielding productive results.

"Maybe … fraud retractions and error retractions are different in ways that are correlated for the potential for follow-on research," Azoulay says.

The researchers tried to account for that in their study, by looking at the number of citations garnered by retracted articles prior to the discovery of problems, then correlating that with the impact on related papers. But the numbers do not show an underlying structural reason why retractions for fraud, as opposed to unintentional mistakes, should generate a larger impact on related prior research.

The researchers have a couple of specific policy suggestions to limit unwarranted damage to the accretion of scientific knowledge. For instance, journals that retract articles should offer detailed explanations of why those papers are no longer valid. It would not be hard, they suggest, to develop a basic system of categories of retractions.

"By lumping together honest mistakes and misconduct, we're undermining the smooth functioning of the retraction system," Azoulay says.