SCIENCEINSIDER

Breaking news and analysis from the world of science policy

Study claims $28 billion a year spent on irreproducible biomedical research

Jocelyn is a staff writer for Sciencemagazine.

Email Jocelyn

Jocelyn Kaiser

9 June 2015 1:30 pm

Comments

An eye-popping $28 billion is spent in the United States each year on preclinical research that can’t be reproduced by other researchers. That’s the conclusion of a provocative analysis published today in part by economists who based it on past studies of error rates in biomedical studies.

Meanwhile, the National Institutes of Health (NIH) today issued new criteria for grant reviews aimed at bolstering the reproducibility of NIH-funded research.

The lead author of the new price tag for reproducibility says it is meant to stimulate discussion. “We’re pointing to the economic cost, but we’re also trying to promote some solutions. That’s really the message of the paper,” says biologist Leonard Freedman, president of the nonprofit Global Biological Standards Institute (GBSI) in Washington, D.C., of the perspective in PLOS Biology.

One expert, however, is skeptical of the $28 billion figure, saying it may overstate the extent of any problem.

To come up with the number, Freedman and economists Iain Cockburn and Timothy Simcoe of Boston University combed the literature for two dozen or so studies that tried to quantify how many biomedical papers are flawed because of specific problems such as a contaminated cell line. Looking across these data, they estimate that 53% of all preclinical studies have errors that mean they are not reproducible. The most common reasons included problems with reagents and reference materials (36%), study design (28%), data analysis and reporting (25%), and laboratory protocols (11%).

The 53% is roughly comparable to a handful of “top-down” studies that tried to reproduce a set of findings, Cockburn says. For example, a widely cited analysis by Amgen found that only 11% of 53 preclinical cancer papers could be reproduced in the company’s labs. Other studies found a rate closer to 50%.

From there, calculating the economic impact was simple. Rounding the 53% to a “conservative” irreproducibility estimate of 50%, the researchers multiplied by the $56 billion a year that NIH and other U.S. public and private funders spend on preclinical research. That yields $28 billion in irreproducible preclinical research.

Does that mean the money is being wasted? Not exactly, the authors say. Instead, they say that addressing irreproducibility would make that funding go much further. “This is a time to invest more, not less, with a relatively small part of that investment to improve the irreproducibility rate,” Freedman says.

Among other solutions, researchers should receive better training in study design, and vendors and scientists should sell and use only validated reagents, the PLOS Biology commentary concludes. GBSI itself is encouraging the research community to put in place accepted standards for authenticating cell lines.

NIH is also taking steps to improve reproducibility. In an announcement today, the agency describes four new criteria that grant reviewers will be asked to consider starting next January. They include the strength of the scientific premise that the proposed study builds on; the rigor of the study’s design; the proposal’s consideration of the sex of cells, research animals, or human subjects; and whether reagents have been authenticated.

Although he hasn’t seen the details of the PLOS Biology commentary, one scientist who has studied why papers are retracted says it “sounds sensationalistic.” Microbiologist Ferric Fang of the University of Washington, Seattle, questions whether it is possible to extrapolate from a few previous studies on reproducibility and conclude that 50% of preclinical research can’t be reproduced. That’s partly because being unable to replicate a specific experiment is not the same as failing to reproduce that finding under different conditions; that outcome is more common but doesn’t mean the original finding was wrong. “To suggest that 50% of research dollars are being wasted is ridiculous and unhelpful,” Fang says.

Posted in