Phoenix Rising supports the Millions Missing global day of protest
Phoenix Rising is delighted to support the demands being made in the ME/CFS community’s first-ever global day of protest …
Discuss the article on the Forums.

New from Nick Brown: how psychology incentivises bad science

Discussion in 'General ME/CFS News' started by Simon, Jun 17, 2016.

  1. Simon


    Monmouth, UK
    At the recent British Psychological Society Replicability and Reproducibility Debate Nick Brown (@sTeamTraen, good science star) presented his views An outsider’s view of the incentive structure of science,with a focus on psychology in particular, which is what I pick up here given psychology's considerable contribution to the mecfs research literature.

    (Not that the mecfs biomedical literature is much to boast about, though things now seem to be improving rapidly).

    Extensive highlights
    [my comments]​

    General incentive problems in science:
    • Funding, Promotions, Tenure
    • Power, Prestige, “Excellence”

    General enablers of bad science
    • Publication bias
    • Peer review is a lottery
    • Journals are chasing after impact factor, Open access journals are chasing after money
    • Publish-or-perish culture
    • Journals are unwilling to publish comments
    • Most published research results are false

    Problems (more or less) specific to psychology [I’ve picked out those that seem most relevant to ME/CFS]

    1. It sells
    Popular media love “Science has shown that people do X” stories
    [focus on symptoms, rest too much and get deconditioned, et voila: a life-destroying illness, perhaps?]

    2. No “proof by technology”
    • The constructs we measure are not externally verifiable [eg fatigue, beliefs]
    • We have very little solid theory: Critical discussion of the relative merits of theories is rare

    3. A lot of psychologists are not very good at statistics
    • Most psychologists do not know how to interpret p values correctly
    • Personal observation: many psychologists consider numbers (at best) a “necessary evil”

    False positives are not a career-limiting issue in psychology
    • Unlike physical and biomedical sciences:
    Fleischmann and Pons (cold fusion)
    Benveniste (homeopathy, memory of water)

    • “Nobody is going to use this to fly a plane”
    • Replication attempts (historically) unlikely
    • Unlimited post hoc “hidden moderators”

    There’s quite a long discussion of fraud/data fabrication which I won’t cover

    Talking about Questionable Research Practices, QRPs, non-fraudulent :

    No disincentives for QRPs

    [Just includig his point which relates to the general issue of incentives for bad science:]
    • Is it necessary to cheat to compete?

    Brown & Heathers, 2016 (preprint paper)
    Looks at basic statistical errors in 260 articles, found many - and many with problems were unwilling to share data.

    We’re running out of parsimonious [simple] explanations
    Maybe one reason we can’t reproduce results is they were reported incorrectly (or fabricated) in the first place

    Why won’t psychological researchers share data?
    • Less than half of requests to share data are successful
    • Most common response: Silence
    • Most datasets pose no real ethical issues
    • Effort to curate [get data in shape for others to understand] ought to be minimal (and should have been done anyway)

    Water flows downhill
    • Psychologists are rational people
    • Rational people will do what the system incentivises them to do
    • If we have QRPs (and fraud), it’s because the system is set up to reward it
    • The upside of cheating is perceived as greater than the downside

    As for reform and introducing good practices:
    • In the absence of a credible enforcement mechanism, incentives trump rules

    [On that happy note, I'll leave you.]

    Nb this is my personal set of highlights, I recommend you read the full presentation (powerpoint). If anything isn't right, Nick, please shout.
  2. A.B.

    A.B. Senior Member

    One interpretation:

    Freud was a fraud (or possibly just very high on cocaine), and got this circus started setting the bar for results so high it could only be met with deliberate or accidental distortion of reality. The field avoids robust methods because they produce null or tiny effects. Instead, they try to achieve the impossible. There are probably some valuable psychological interventions too but they are drowned in crap.
  3. sTeamTraen


    Simon, this is fine by me. :)

    A.B.: Most psychologists reject everything Freud ever said, for all kinds of reasons, notably because of his circular reasoning and lack of falsifiability. What I always find amusing, however, is how psychologists' own behaviour is so Freudian in so many ways...
    Snow Leopard, Valentijn and Simon like this.
  4. A.B.

    A.B. Senior Member

    What has changed besides the narrative? The central idea that there are two types of illnesses, the ones that originate from biological factors, and the ones that originate from thoughts and emotions hasn't changed. Is there a lab experiment that has shown the second category even exists? Where is the skilled psychologist that has, for example, experimentally induced IBS in himself with mere thoughts and emotions? Is that too physical? What about serious depression? If it can't be proven that these pathways lead to illness maybe it's time to accept that it's a null field. Why are these ideas accepted as truth without any real evidence?
    Last edited: Jun 17, 2016
    Keela Too, panckage, Comet and 7 others like this.
  5. Woolie

    Woolie Senior Member

    @sTeamTraen, I think A.B. knows this. The point here is simply that Freud's demonstrations suggested psychological variables can have outrageously powerful effects on all kinds of behaviours, including illness. This led to a culture where it was believed that "good" psychology involves generating impressive results. Such work is rewarded over more carefully designed work that yields more equivocal results.

    And let's not get started on Eysenck...
    Solstice, Snow Leopard, Simon and 2 others like this.
  6. Daisymay

    Daisymay Senior Member

    Here, here!
  7. BurnA

    BurnA Senior Member

    I think the above sounds like it is making excuses for or rationalizing the psychologists engaging in QRP or fraud.

    There are plenty of psychologists engaging in high quality scientific research, because ultimately they consider themselves part of a scientific domain and that is how science works. The system rewards them just as much. The mightnt get the same headline grabbing attention because they aren't making their startling results up, but they are still being rewarded within the system the choose to belong.

    The problem isn't the system, its individuals within the system. Individuals who choose to engage in QRP or fraud, individuals who dont speak out about bad practises, individuals who reward bad science and the individuals who created the system.
    Snow Leopard likes this.
  8. Snow Leopard

    Snow Leopard Hibernating

    South Australia
    Is there generally an attitude amongst researchers that their own shit doesn't stink? It's always other groups of people?

    I saw this attitude amongst the field I studied in (Chemistry), when I talked about QRP, most seemed to imply it wasn't a problem in Chemistry and it was just a problem for all those social scientists or something. Maybe chemists are all saints, but I doubt it. (I've heard of lost of sloppiness/QRP in cellular/molecular biology too!)
  9. Woolie

    Woolie Senior Member

    I agree a bit and disagree a bit too. I agree with you about cheating. But most QRPs involve researchers who would never consider cheating.

    A lot of QRPs happen because researchers believe in their hypotheses, so they look extra hard for confirmation, and aren't oriented to disconfirming evidence. Richard Feynman's quote might be aprt here "The first principle is that you must not fool yourself — and you are the easiest person to fool."

    Of course, needing to have X publications may be a strong unconscious motive for finding that positive result, Im not saying motive does not operate, but in most cases, there's no actual "cheating"

    Journals contribute by preferring positive results. But even this tendency is unlikely to be outright self interest. The way we set up our hypothesis, a failure to reject the null hypothesis is kind of hard to interpret. It can happen for so many reasons. So people believed they weren't much use in advancing science. Only recently have people begun pointing out how this bias can impact on a field of knowledge in the long term.

    I think Nick Brown's article underestimates these "innocent" biases and overestiamtes actual self-interest.
    Simon and Snow Leopard like this.
  10. chipmunk1

    chipmunk1 Senior Member

    Freud's mentor Charcot believed that in some people with a certain genetic predisposition, Hysteria could be triggered by a physical trauma, an accident for example. Not being able to identify these traumas in all of his patient's life histories Freud took it one step further and claimed that a non-physical(psychological) trauma. caused Hysteria. Before Freud a trauma was referring to a physical injury, today in our culture most people think of a psychological trauma.

    Anything that attempts to treat any physical or mental problem by talking about your childhood is rooted in freudian thought.

    Psychoanalysis operated more like a cult than a science. In fact just wanting to study Psychoanalysis scientifically was seen as some kind of emotional disturbance.

    Thus for at least the first half of the past century, most talk therapy resembled a religion.

    It was never investigated whether or not Psychoanalysis was just an expensive placebo. Therapists had a rule of thumb that one third of their patients would improve, one third stayed the same and one third got worse. Most likely this was the natural course of the conditions they were treating.

    It was simply accepted that talk therapy could successfully treat various mental and physical illnesses. It was forbidden to test it in controlled trials though.

    Many in the field probably knew that their talk therapy was not very effective. Even Freud himself admitted privately that he could not help his clients. Lacan was quite open about this fact in his later years.

    Making unproven claims is part of the history of talk therapy.

    Psychoanalysis created demand and a mainstream market for talking cures. Once the demand is there someone will step in and offer the service regardless of efficacy.
    TiredSam, sarah darwins and A.B. like this.
  11. A.B.

    A.B. Senior Member

    Probably also that students are taught flawed methods from the start. See the online surveys about CFS made by psychology students that are typically riddled with problems. Students could make a better contribution by summarizing interviews with patients. The survey is probably more attractive because of a gloss of science but this conveys a false sense of reliability.

    For a real world example, see the recent paper by Crawley where pediatric CFS could be diagnosed if "mom said he has it".

    I've also heard of an occupational therapist diagnosing CFS over the phone.

    Clearly these people don't treat the topic with respect. More like a mental playground where paper after paper is churned out with no regard for patient welfare or reality, the conclusions always supporting the perpetuation of these practices.
    Last edited: Jun 18, 2016
    panckage, Comet, Valentijn and 3 others like this.
  12. panckage

    panckage Senior Member

    Vancouver, BC
    I believe this is a harsh understatement. I took an undergrad psychology stats course (easy mark haha) at the University of British Columbia, ranked the #34 best University in the world last year. My professor taught an ENTIRE stats course without doing a SINGLE problem. The lectures were 100% PowerPoint. I didn't mind as I already knew the material and the teacher was really funny (she was like a boisterous female comedian who made a lot of oral sex jokes. I was one of the few in the class that found them funny. Bunch of uptight stats-illiterate wanna be SJW types that class was.... :rofl:). The other kids in the class were really confused. I ended up helping a lot of them. All the tests were multiple choice as well, no written sections if I remember correctly. No wonder psych is so pathetic. I even remember hearing psych grad students talking in the library that they shouldn't have to learn statistics, that they should just pay people to do that. Oh God I didn't say anything but that was disgusting...

    Not sure how widespread this issue is but from the social 'science' crud that dominates the headlines my experience is probably not uncommon

    OTOH I did date a psychology professor in the field of positive thinking(?) and she was amazing! We never discussed stats though and she never showed me her work :heart::rofl:
    Cheshire and BurnA like this.

See more popular forum discussions.

Share This Page