Simon
Senior Member
- Messages
- 3,789
- Location
- Monmouth, UK
At the recent British Psychological Society Replicability and Reproducibility Debate Nick Brown (@sTeamTraen, good science star) presented his views An outsider’s view of the incentive structure of science,with a focus on psychology in particular, which is what I pick up here given psychology's considerable contribution to the mecfs research literature.
(Not that the mecfs biomedical literature is much to boast about, though things now seem to be improving rapidly).
General incentive problems in science:
• Funding, Promotions, Tenure
• Power, Prestige, “Excellence”
General enablers of bad science
• Publication bias
• Peer review is a lottery
• Journals are chasing after impact factor, Open access journals are chasing after money
• Publish-or-perish culture
• Journals are unwilling to publish comments
• Most published research results are false
Problems (more or less) specific to psychology [I’ve picked out those that seem most relevant to ME/CFS]
1. It sells
Popular media love “Science has shown that people do X” stories
[focus on symptoms, rest too much and get deconditioned, et voila: a life-destroying illness, perhaps?]
2. No “proof by technology”
• The constructs we measure are not externally verifiable [eg fatigue, beliefs]
• We have very little solid theory: Critical discussion of the relative merits of theories is rare
3. A lot of psychologists are not very good at statistics
• Most psychologists do not know how to interpret p values correctly
• Personal observation: many psychologists consider numbers (at best) a “necessary evil”
False positives are not a career-limiting issue in psychology
• Unlike physical and biomedical sciences:
Fleischmann and Pons (cold fusion)
Benveniste (homeopathy, memory of water)
Reasons:
• “Nobody is going to use this to fly a plane”
• Replication attempts (historically) unlikely
• Unlimited post hoc “hidden moderators”
There’s quite a long discussion of fraud/data fabrication which I won’t cover
Talking about Questionable Research Practices, QRPs, non-fraudulent :
No disincentives for QRPs
[Just includig his point which relates to the general issue of incentives for bad science:]
• Is it necessary to cheat to compete?
Brown & Heathers, 2016 (preprint paper)
Looks at basic statistical errors in 260 articles, found many - and many with problems were unwilling to share data.
We’re running out of parsimonious [simple] explanations
Maybe one reason we can’t reproduce results is they were reported incorrectly (or fabricated) in the first place
Why won’t psychological researchers share data?
• Less than half of requests to share data are successful
• Most common response: Silence
• Most datasets pose no real ethical issues
• Effort to curate [get data in shape for others to understand] ought to be minimal (and should have been done anyway)
Water flows downhill
• Psychologists are rational people
• Rational people will do what the system incentivises them to do
• If we have QRPs (and fraud), it’s because the system is set up to reward it
• The upside of cheating is perceived as greater than the downside
As for reform and introducing good practices:
• In the absence of a credible enforcement mechanism, incentives trump rules
[On that happy note, I'll leave you.]
Nb this is my personal set of highlights, I recommend you read the full presentation (powerpoint). If anything isn't right, Nick, please shout.
(Not that the mecfs biomedical literature is much to boast about, though things now seem to be improving rapidly).
Extensive highlights
[my comments]
[my comments]
General incentive problems in science:
• Funding, Promotions, Tenure
• Power, Prestige, “Excellence”
General enablers of bad science
• Publication bias
• Peer review is a lottery
• Journals are chasing after impact factor, Open access journals are chasing after money
• Publish-or-perish culture
• Journals are unwilling to publish comments
• Most published research results are false
Problems (more or less) specific to psychology [I’ve picked out those that seem most relevant to ME/CFS]
1. It sells
Popular media love “Science has shown that people do X” stories
[focus on symptoms, rest too much and get deconditioned, et voila: a life-destroying illness, perhaps?]
2. No “proof by technology”
• The constructs we measure are not externally verifiable [eg fatigue, beliefs]
• We have very little solid theory: Critical discussion of the relative merits of theories is rare
3. A lot of psychologists are not very good at statistics
• Most psychologists do not know how to interpret p values correctly
• Personal observation: many psychologists consider numbers (at best) a “necessary evil”
False positives are not a career-limiting issue in psychology
• Unlike physical and biomedical sciences:
Fleischmann and Pons (cold fusion)
Benveniste (homeopathy, memory of water)
Reasons:
• “Nobody is going to use this to fly a plane”
• Replication attempts (historically) unlikely
• Unlimited post hoc “hidden moderators”
There’s quite a long discussion of fraud/data fabrication which I won’t cover
Talking about Questionable Research Practices, QRPs, non-fraudulent :
No disincentives for QRPs
[Just includig his point which relates to the general issue of incentives for bad science:]
• Is it necessary to cheat to compete?
Brown & Heathers, 2016 (preprint paper)
Looks at basic statistical errors in 260 articles, found many - and many with problems were unwilling to share data.
We’re running out of parsimonious [simple] explanations
Maybe one reason we can’t reproduce results is they were reported incorrectly (or fabricated) in the first place
Why won’t psychological researchers share data?
• Less than half of requests to share data are successful
• Most common response: Silence
• Most datasets pose no real ethical issues
• Effort to curate [get data in shape for others to understand] ought to be minimal (and should have been done anyway)
Water flows downhill
• Psychologists are rational people
• Rational people will do what the system incentivises them to do
• If we have QRPs (and fraud), it’s because the system is set up to reward it
• The upside of cheating is perceived as greater than the downside
As for reform and introducing good practices:
• In the absence of a credible enforcement mechanism, incentives trump rules
[On that happy note, I'll leave you.]
Nb this is my personal set of highlights, I recommend you read the full presentation (powerpoint). If anything isn't right, Nick, please shout.