The Atlantic: "The Inevitable Evolution of Bad Science"

[Jennifer J]

http://www.theatlantic.com/science/archive/2016/09/the-inevitable-evolution-of-bad-science/500609/

The problem, as others have noted, is that what is good for individual scientists is not necessarily what is good for science as a whole. A scientist’s career currently depends on publishing as many papers as possible in the most prestigious possible journals. More than any other metric, that’s what gets them prestige, grants, and jobs.

Now, imagine you’re a researcher who wants to game this system. Here’s what you do. Run many small and statistically weak studies. Tweak your methods on the fly to ensure positive results. If you get negative results, sweep them under the rug. Never try to check old results; only pursue new and exciting ones. These are not just flights of fancy. We know that such practices abound. They’re great for getting publications, but they also pollute the scientific record with results that aren’t actually true. As Richard Horton, editor of The Lancet once wrote, “No one is incentivized to be right. Instead, scientists are incentivized to be productive.”

He and McElreath demonstrated this by creating a mathematical model in which simulated labs compete with each other and evolve—think SimAcademia. The labs choose things to study, run experiments to test their hypotheses, and try to publish their results. They vary in how much effort they expend in testing their ideas, which affects how many results they get, and how reliable those results are. There’s a trade-off: more effort means truer but fewer publications.

In the model, as in real academia, positive results are easier to publish than negative one, and labs that publish more get more prestige, funding, and students. They also pass their practices on. With every generation, one of the oldest labs dies off, while one of the most productive one reproduces, creating an offspring that mimics the research style of the parent. That’s the equivalent of a student from a successful team starting a lab of their own.

Munafo is hopeful. “We have moved on from describing the problem to understanding its nature,” he says. “This is a healthy sign. Hopefully it will lead to clues as to where we can most efficiently change incentive structures. We're in the middle of a fascinating natural experiment, with lots of innovations being introduced or piloted. What works and doesn't work, and what is popular versus unpopular, remains to be seen.”

“I don’t want to be overly pessimistic,” says Smaldino. “There are a lot of really high-quality scientists who strive to do high-quality work. There are tons of individuals realize that quality matters. I just hope that sentiment prevails.”