• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

New from Nick Brown: how psychology incentivises bad science

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK
At the recent British Psychological Society Replicability and Reproducibility Debate Nick Brown (@sTeamTraen, good science star) presented his views An outsider’s view of the incentive structure of science,with a focus on psychology in particular, which is what I pick up here given psychology's considerable contribution to the mecfs research literature.

(Not that the mecfs biomedical literature is much to boast about, though things now seem to be improving rapidly).

Extensive highlights
[my comments]​

General incentive problems in science:
• Funding, Promotions, Tenure
• Power, Prestige, “Excellence”

General enablers of bad science
• Publication bias
• Peer review is a lottery
• Journals are chasing after impact factor, Open access journals are chasing after money
• Publish-or-perish culture
• Journals are unwilling to publish comments
• Most published research results are false

Problems (more or less) specific to psychology [I’ve picked out those that seem most relevant to ME/CFS]

1. It sells
Popular media love “Science has shown that people do X” stories
[focus on symptoms, rest too much and get deconditioned, et voila: a life-destroying illness, perhaps?]

2. No “proof by technology”
• The constructs we measure are not externally verifiable [eg fatigue, beliefs]
• We have very little solid theory: Critical discussion of the relative merits of theories is rare

3. A lot of psychologists are not very good at statistics
• Most psychologists do not know how to interpret p values correctly
• Personal observation: many psychologists consider numbers (at best) a “necessary evil”

False positives are not a career-limiting issue in psychology
• Unlike physical and biomedical sciences:
Fleischmann and Pons (cold fusion)
Benveniste (homeopathy, memory of water)


Reasons:
• “Nobody is going to use this to fly a plane”
• Replication attempts (historically) unlikely
• Unlimited post hoc “hidden moderators”

There’s quite a long discussion of fraud/data fabrication which I won’t cover

Talking about Questionable Research Practices, QRPs, non-fraudulent :

No disincentives for QRPs

[Just includig his point which relates to the general issue of incentives for bad science:]
• Is it necessary to cheat to compete?

Brown & Heathers, 2016 (preprint paper)
Looks at basic statistical errors in 260 articles, found many - and many with problems were unwilling to share data.

We’re running out of parsimonious [simple] explanations
Maybe one reason we can’t reproduce results is they were reported incorrectly (or fabricated) in the first place

Why won’t psychological researchers share data?
• Less than half of requests to share data are successful
• Most common response: Silence
• Most datasets pose no real ethical issues
• Effort to curate [get data in shape for others to understand] ought to be minimal (and should have been done anyway)

Water flows downhill
• Psychologists are rational people
• Rational people will do what the system incentivises them to do
• If we have QRPs (and fraud), it’s because the system is set up to reward it
• The upside of cheating is perceived as greater than the downside

As for reform and introducing good practices:
• In the absence of a credible enforcement mechanism, incentives trump rules

[On that happy note, I'll leave you.]

Nb this is my personal set of highlights, I recommend you read the full presentation (powerpoint). If anything isn't right, Nick, please shout.
 

A.B.

Senior Member
Messages
3,780
One interpretation:

Freud was a fraud (or possibly just very high on cocaine), and got this circus started setting the bar for results so high it could only be met with deliberate or accidental distortion of reality. The field avoids robust methods because they produce null or tiny effects. Instead, they try to achieve the impossible. There are probably some valuable psychological interventions too but they are drowned in crap.
 
Messages
7
Simon, this is fine by me. :)

A.B.: Most psychologists reject everything Freud ever said, for all kinds of reasons, notably because of his circular reasoning and lack of falsifiability. What I always find amusing, however, is how psychologists' own behaviour is so Freudian in so many ways...
 

A.B.

Senior Member
Messages
3,780
Most psychologists reject everything Freud ever said, for all kinds of reasons, notably because of his circular reasoning and lack of falsifiability.

What has changed besides the narrative? The central idea that there are two types of illnesses, the ones that originate from biological factors, and the ones that originate from thoughts and emotions hasn't changed. Is there a lab experiment that has shown the second category even exists? Where is the skilled psychologist that has, for example, experimentally induced IBS in himself with mere thoughts and emotions? Is that too physical? What about serious depression? If it can't be proven that these pathways lead to illness maybe it's time to accept that it's a null field. Why are these ideas accepted as truth without any real evidence?
 
Last edited:
Messages
3,263
A.B.: Most psychologists reject everything Freud ever said, for all kinds of reasons, notably because of his circular reasoning and lack of falsifiability. What I always find amusing, however, is how psychologists' own behaviour is so Freudian in so many ways...
@sTeamTraen, I think A.B. knows this. The point here is simply that Freud's demonstrations suggested psychological variables can have outrageously powerful effects on all kinds of behaviours, including illness. This led to a culture where it was believed that "good" psychology involves generating impressive results. Such work is rewarded over more carefully designed work that yields more equivocal results.

And let's not get started on Eysenck...
 

Daisymay

Senior Member
Messages
754
What has changed besides the narrative? The central idea that there are two types of illnesses, the ones that originate from biological factors, and the ones that originate from thoughts and emotions hasn't changed. Is there a lab experiment that has shown the second category even exists? Where is the skilled psychologist that has, for example, experimentally induced IBS in himself with mere thoughts and emotions? Is that too physical? What about serious depression? If it can't be proven that these pathways lead to illness maybe it's time to accept that it's a null field. Why are these ideas accepted as truth without any real evidence?

Here, here!
 
Messages
2,087
Water flows downhill
• Psychologists are rational people
• Rational people will do what the system incentivises them to do
• If we have QRPs (and fraud), it’s because the system is set up to reward it
• The upside of cheating is perceived as greater than the downside

I think the above sounds like it is making excuses for or rationalizing the psychologists engaging in QRP or fraud.

There are plenty of psychologists engaging in high quality scientific research, because ultimately they consider themselves part of a scientific domain and that is how science works. The system rewards them just as much. The mightnt get the same headline grabbing attention because they aren't making their startling results up, but they are still being rewarded within the system the choose to belong.

The problem isn't the system, its individuals within the system. Individuals who choose to engage in QRP or fraud, individuals who dont speak out about bad practises, individuals who reward bad science and the individuals who created the system.
 

Snow Leopard

Hibernating
Messages
5,902
Location
South Australia
A.B.: Most psychologists reject everything Freud ever said, for all kinds of reasons, notably because of his circular reasoning and lack of falsifiability. What I always find amusing, however, is how psychologists' own behaviour is so Freudian in so many ways...

Is there generally an attitude amongst researchers that their own shit doesn't stink? It's always other groups of people?

I saw this attitude amongst the field I studied in (Chemistry), when I talked about QRP, most seemed to imply it wasn't a problem in Chemistry and it was just a problem for all those social scientists or something. Maybe chemists are all saints, but I doubt it. (I've heard of lost of sloppiness/QRP in cellular/molecular biology too!)
 
Messages
3,263
The problem isn't the system, its individuals within the system. Individuals who choose to engage in QRP or fraud, individuals who dont speak out about bad practises, individuals who reward bad science and the individuals who created the system.
I agree a bit and disagree a bit too. I agree with you about cheating. But most QRPs involve researchers who would never consider cheating.

A lot of QRPs happen because researchers believe in their hypotheses, so they look extra hard for confirmation, and aren't oriented to disconfirming evidence. Richard Feynman's quote might be aprt here "The first principle is that you must not fool yourself — and you are the easiest person to fool."

Of course, needing to have X publications may be a strong unconscious motive for finding that positive result, Im not saying motive does not operate, but in most cases, there's no actual "cheating"

Journals contribute by preferring positive results. But even this tendency is unlikely to be outright self interest. The way we set up our hypothesis, a failure to reject the null hypothesis is kind of hard to interpret. It can happen for so many reasons. So people believed they weren't much use in advancing science. Only recently have people begun pointing out how this bias can impact on a field of knowledge in the long term.

I think Nick Brown's article underestimates these "innocent" biases and overestiamtes actual self-interest.
 

chipmunk1

Senior Member
Messages
765
What has changed besides the narrative? The central idea that there are two types of illnesses, the ones that originate from biological factors, and the ones that originate from thoughts and emotions hasn't changed. Is there a lab experiment that has shown the second category even exists? Where is the skilled psychologist that has, for example, experimentally induced IBS in himself with mere thoughts and emotions? Is that too physical? What about serious depression? If it can't be proven that these pathways lead to illness maybe it's time to accept that it's a null field. Why are these ideas accepted as truth without any real evidence?

Freud's mentor Charcot believed that in some people with a certain genetic predisposition, Hysteria could be triggered by a physical trauma, an accident for example. Not being able to identify these traumas in all of his patient's life histories Freud took it one step further and claimed that a non-physical(psychological) trauma. caused Hysteria. Before Freud a trauma was referring to a physical injury, today in our culture most people think of a psychological trauma.

Anything that attempts to treat any physical or mental problem by talking about your childhood is rooted in freudian thought.

Psychoanalysis operated more like a cult than a science. In fact just wanting to study Psychoanalysis scientifically was seen as some kind of emotional disturbance.

Thus for at least the first half of the past century, most talk therapy resembled a religion.

It was never investigated whether or not Psychoanalysis was just an expensive placebo. Therapists had a rule of thumb that one third of their patients would improve, one third stayed the same and one third got worse. Most likely this was the natural course of the conditions they were treating.

It was simply accepted that talk therapy could successfully treat various mental and physical illnesses. It was forbidden to test it in controlled trials though.

Many in the field probably knew that their talk therapy was not very effective. Even Freud himself admitted privately that he could not help his clients. Lacan was quite open about this fact in his later years.

Making unproven claims is part of the history of talk therapy.

Psychoanalysis created demand and a mainstream market for talking cures. Once the demand is there someone will step in and offer the service regardless of efficacy.
 

A.B.

Senior Member
Messages
3,780
A lot of QRPs happen because researchers believe in their hypotheses, so they look extra hard for confirmation, and aren't oriented to disconfirming evidence.

Probably also that students are taught flawed methods from the start. See the online surveys about CFS made by psychology students that are typically riddled with problems. Students could make a better contribution by summarizing interviews with patients. The survey is probably more attractive because of a gloss of science but this conveys a false sense of reliability.

For a real world example, see the recent paper by Crawley where pediatric CFS could be diagnosed if "mom said he has it".

I've also heard of an occupational therapist diagnosing CFS over the phone.

Clearly these people don't treat the topic with respect. More like a mental playground where paper after paper is churned out with no regard for patient welfare or reality, the conclusions always supporting the perpetuation of these practices.
 
Last edited:

panckage

Senior Member
Messages
777
Location
Vancouver, BC
3. A lot of psychologists are not very good at statistics
• Most psychologists do not know how to interpret p values correctly
• Personal observation: many psychologists consider numbers (at best) a “necessary evil”
I believe this is a harsh understatement. I took an undergrad psychology stats course (easy mark haha) at the University of British Columbia, ranked the #34 best University in the world last year. My professor taught an ENTIRE stats course without doing a SINGLE problem. The lectures were 100% PowerPoint. I didn't mind as I already knew the material and the teacher was really funny (she was like a boisterous female comedian who made a lot of oral sex jokes. I was one of the few in the class that found them funny. Bunch of uptight stats-illiterate wanna be SJW types that class was.... :rofl:). The other kids in the class were really confused. I ended up helping a lot of them. All the tests were multiple choice as well, no written sections if I remember correctly. No wonder psych is so pathetic. I even remember hearing psych grad students talking in the library that they shouldn't have to learn statistics, that they should just pay people to do that. Oh God I didn't say anything but that was disgusting...

Not sure how widespread this issue is but from the social 'science' crud that dominates the headlines my experience is probably not uncommon

OTOH I did date a psychology professor in the field of positive thinking(?) and she was amazing! We never discussed stats though and she never showed me her work :heart::rofl: