http://io9.com/i-fooled-millions-into-thinking-chocolate-helps-weight-1707251800
Amazing.
A journalist basically pulled a long con on the media, demonstrating some of the issues plaguing science reporting.
He actually ran a study, one that was intentionally methodologically flawed, then p-hacked and spun his results, submitted to pay-to-publish journals, and then enthusiastically promoted his published article. And it worked.
Here are some excerpts:
"I am Johannes Bohannon, Ph.D. Well, actually my name is John, and I’m a journalist. I do have a Ph.D., but it’s in the molecular biology of bacteria, not humans. The Institute of Diet and Health? That’s nothing more than a website.
Other than those fibs, the study was 100 percent authentic. My colleagues and I recruited actual human subjects in Germany. We ran an actual clinical trial, with subjects randomly assigned to different diet regimes. And the statistically significant benefits of chocolate that we reported are based on the actual data. It was, in fact, a fairly typical study for the field of diet research. Which is to say: It was terrible science. The results are meaningless, and the health claims that the media blasted out to millions of people around the world are utterly unfounded."
" The only problem with the diet science beat is that it’s science. You have to know how to read a scientific paper—and actually bother to do it. For far too long, the people who cover this beat have treated it like gossip, echoing whatever they find in press releases. Hopefully our little experiment will make reporters and readers alike more skeptical.
If a study doesn’t even list how many people took part in it, or makes a bold diet claim that’s “statistically significant” but doesn’t say how big the effect size is, you should wonder why. But for the most part, we don’t. Which is a pity, because journalists are becoming the de facto peer review system. And when we fail, the world is awash in junk science.
There was one glint of hope in this tragicomedy. While the reporters just regurgitated our “findings,” many readers were thoughtful and skeptical. In the online comments, they posed questions that the reporters should have asked.
“Why are calories not counted on any of the individuals?” asked a reader on a bodybuilding forum. “The domain [for the Institute of Diet and Health web site] was registered at the beginning of March, and dozens of blogs and news magazines (see Google) spread this study without knowing what or who stands behind it,” said a reader beneath the story in Focus, one of Germany’s leading online magazines.
Or as one prescient reader of the 4 April story in the Daily Express put it, “Every day is April Fool’s in nutrition.” "
Amazing.
A journalist basically pulled a long con on the media, demonstrating some of the issues plaguing science reporting.
He actually ran a study, one that was intentionally methodologically flawed, then p-hacked and spun his results, submitted to pay-to-publish journals, and then enthusiastically promoted his published article. And it worked.
Here are some excerpts:
"I am Johannes Bohannon, Ph.D. Well, actually my name is John, and I’m a journalist. I do have a Ph.D., but it’s in the molecular biology of bacteria, not humans. The Institute of Diet and Health? That’s nothing more than a website.
Other than those fibs, the study was 100 percent authentic. My colleagues and I recruited actual human subjects in Germany. We ran an actual clinical trial, with subjects randomly assigned to different diet regimes. And the statistically significant benefits of chocolate that we reported are based on the actual data. It was, in fact, a fairly typical study for the field of diet research. Which is to say: It was terrible science. The results are meaningless, and the health claims that the media blasted out to millions of people around the world are utterly unfounded."
" The only problem with the diet science beat is that it’s science. You have to know how to read a scientific paper—and actually bother to do it. For far too long, the people who cover this beat have treated it like gossip, echoing whatever they find in press releases. Hopefully our little experiment will make reporters and readers alike more skeptical.
If a study doesn’t even list how many people took part in it, or makes a bold diet claim that’s “statistically significant” but doesn’t say how big the effect size is, you should wonder why. But for the most part, we don’t. Which is a pity, because journalists are becoming the de facto peer review system. And when we fail, the world is awash in junk science.
There was one glint of hope in this tragicomedy. While the reporters just regurgitated our “findings,” many readers were thoughtful and skeptical. In the online comments, they posed questions that the reporters should have asked.
“Why are calories not counted on any of the individuals?” asked a reader on a bodybuilding forum. “The domain [for the Institute of Diet and Health web site] was registered at the beginning of March, and dozens of blogs and news magazines (see Google) spread this study without knowing what or who stands behind it,” said a reader beneath the story in Focus, one of Germany’s leading online magazines.
Or as one prescient reader of the 4 April story in the Daily Express put it, “Every day is April Fool’s in nutrition.” "
Last edited: