Why We Find And Expose Bad Science (food science)

[RogerBlack]

https://medium.com/@jamesheathers/why-we-find-and-expose-bad-science-e47387a0e333

If you don’t have time to read it, 50 word summary:

• a lab published a blog post outlining terrible research practices
• subsequently, we wondered if there were errors in their work
• there were lots (45), so we pointed them out (it took ages)
• so far, 3 papers retracted, 8 corrected, more soon
• the situation has gone on and on and on

(Note: not 45 errors, 45 papers with errors.)

Notable similarities - the researchers involved are making comments about it being all personal, and all the errors being 'minor'.

 

[TiredSam]

Scientific criticism (also occasionally just called ‘science’) is cyber-bullying, apparently.

Sounds familiar. Very good article, makes some good points very relevant to our situation.

 

[Woolie]

This article is so relevant to the issues we have been facing trying to get our voices heard.

Its a story about the nutrition research headed by a guy called Wansink. The writer (James Heathers) and several others found evidence of serious errors and dodgy practices in this team's work. Wansink counterattacked by accusing them of cuber bullying, and suggested they may be trying to topple a public initiative started by him and his mates, which has something to do with improving school lunches.

Other terms Wansink and co used to smear their critics were "shameless little bullies" and "destructo-critics". "Instead of using their skills for good, to help people do better work, rather many of [sic] used their skills to tear people down".

Heathers' response is powerful.

IT’S NOT ALL ABOUT YOU
Do you know who I’m thinking of when we are making criticisms that could affect someone’s career, someone’s professional life?

I’m thinking of the hordes of grad students who never had a chance because they wouldn’t fudge results for a dicey PI...

I’m thinking of everyone who started off curious and intelligent, and ended up mired in an unsustainable system where they couldn’t pursue good ideas because they heard ‘we don’t have the money’ everywhere they turned.

(What money? Government and philanthropic money, used to find out stuff and then give the ideas away. There’s only so much of this money, and there’s less than there used to be.)

So. Bad research just doesn’t affect the people in the area around it, the people who might spend years trying to take a dodgy result and extend it.

It affects everyone else who needs the money.

It’s easier to do bad research. It’s easier not to be careful. Slop the numbers around until something works, fudge a few figures, conveniently misplace a few measurements, and then you’ll be able to say you’ve made discoveries. Dress it all up in pompous gibberish. Call a spade a ‘neo-classical earth-inversion mechanism’. Parlay your amazing ‘discoveries’ into requests for more money...

Before you know it, you’ve built an extremely expensive house of extremely cheap cards.

In this context, when you’re pulled up for doing something wrong (or, really, hundreds of things wrong) the thought “why are they picking on me?” isn’t just short-sighted, it’s deeply selfish.

Of course, bad research is selfish in general — you’re not just prioritising your own needs over even the vaguest concept of the public good, you’re going to screw everyone associated with you. If you do bad research, and this gets pointed out at great volume, then everyone who so much as knows where your office is gets it in the neck...

The public get a raw deal, as usual — you are spending their money producing information they can’t use. Of course, the public do not have access to scientific papers anyway because academic publishing is a sullen trash fire, but that’s beside the point.

Other scientists in your field, of course, do not get off easy. As outlined above, they might have previously tried their own experiments based on the ideas you magicked into being. They wrack their brains reconciling your work. They might get lucky — perhaps your ideas pan out, and your bad papers actually have results that stand up. The point is, of course, that you’ve made them take a chance on that, rather than working from a base of information they can trust.

And science in general, of course, is polluted. More things to read, more cheap ideas thrown around like crumpled chip packets. Another ego demanding a place in public life.

For literally everyone else involved, I have more sympathy. You let them all down. No amount of messenger-shooting will change that.

Another defence Wansink and co raise is that “These problems sound like they’re endemic. Why are you analysing papers instead of criticising scientific culture as a whole?” Heathers replies thus:

BECAUSE WE’VE TRIED THAT
We’ve criticized scientific publication habits, and closed data culture, and bad behaviour in the abstract from arsehole to chapstick. Everyone’s lips are cyanosed from talking until their blood oxygen bottoms out, and their fingers are worn down to little stumpy single-joint Trump-digits from typing out objections. We’ve been criticizing a lot. We’ve all been talking at great volume about systematic problems in science since forever.

The problem with criticizing something in the abstract is that no-one ever thinks the criticism is about them. We can ALL stand up and pound our chests about how things should be, and then go home and participate in the same rubbish research practices at the center of what we decry.

The article then reports how the dodgy researchers constantly minimised the seriousness of their errors. Wansink and co said:
“Science is messy in a lot of ways,”
"...most errors stemmed from “missing data, rounding errors, and [some numbers] being off by 1 or 2.”
"...All the numbers seem to be within one baby carrot of each other. Still, if we can track this data down, we’ll be able to see if this was due to recording, rounding, or measurement"

According to Heathers:

(Wansink had) also discovered that portions of some of his older papers had been republished elsewhere, he said, and had informed six journals of these duplications…Numbers had been missing and statistical calculations off, Wansink wrote, but most importantly……he said he’d realized that some of the data entries for other papers were duplicates or “mismatched.” He suggested that they contact the journals to tell them they were aware of the problems and were going to reanalyze everything. Wansink and his coauthors admitted to having incorrectly described the experiment’s design and number of students involved, used an “inadequate” data analysis method, and mislabeled the graph.

Heathers wonders:

Here’s a question: what if every single one of the criticisms made here is exactly right in every detail? Will ‘happily, I guess my entire body of research is only A BIT wrong’ still be the defense?

 

[Sean]

Before you know it, you’ve built an extremely expensive house of extremely cheap cards.

It's déjà vu all over again.

 

[TiredSam]

IT’S NOT ALL ABOUT YOU

I did like this. It applies not only to Psycho Social Club members such as Wessley and Crawley, who are unable to see criticism of their work as anything other than a personal attack, but also to some proponents of some therapies who take personal offence when their views aren't simply accepted without question.

 

[arewenearlythereyet]

Goes to show that there are always some rogues who don't like following the rules that everyone else uses.

Nutrition is the arse end of food science, where much pseudoscience thrives (a bit like psychiatry and some areas of neurology in medicine)...seems like the pattern is

Not very good academically, hates following rules, has a very large overinflated ego...therefore choose a soft option to specialise in.

Criteria for soft option: complicated subject, very little science, lots of unproven theories and opinions, attractive to politicians wanting to save money.

Defence mechanisms: we don't know everything, criticising what I do means you are just being mean, manipulate media and politicians to prop up lack of skill.

These are techniques used in politics and management generally. It's interesting that the pattern is so similar ....almost like what's going on here at PR at the moment?

 

[Sean]

I did like this. It applies not only to Psycho Social Club members such as Wessley and Crawley, who are unable to see criticism of their work as anything other than a personal attack, but also to some proponents of some therapies who take personal offence when their views aren't simply accepted without question.

The likes of Wessely and Crawley inevitably made it about themselves by refusing to take heed of their critics. That means the problem is also them, not just their ideas.

 

[IreneF]

I sense more than meets the eye with W. and C. They are not just being asswipes because god made them that way. The likeliest explanation is money. The others are marriage to one's ideas and a feeling of superiority.

Anyway, mindlessly slagging them is just flogging a dead horse, as well as providing them with ammunition. Criticize their work, not them. Don't use their names unless referring to a paper.