The control group is out of control.

[Sean]

General article on problems with methodology in science. Can't say I understand it all, but it is clearly of some relevance to our situation.

http://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/

 

[wdb]

I like the reference to parapsychology

Parapsychologists are constantly protesting that they are playing by all the standard scientific rules, and yet their results are being ignored – that they are unfairly being held to higher standards than everyone else. I’m willing to believe that. It just means that the standard statistical methods of science are so weak and flawed as to permit a field of study to sustain itself in the complete absence of any subject matter.

I've never understood why if a parapsychologist conducts an unblinded subjective trial with positive results it's methodologically flawed worthless nonsense but if a psychologist conducts an unblinded subjective trial with positive results it's wonderful research that should be widely and uncritically reported.

 

[A.B.]

Because psychiatry and psychology have managed to convince the world that they are a science, experts and know what they're doing even without appropriate methodology.

 

[chipmunk1]

I like the reference to parapsychology

I've never understood why if a parapsychologist conducts an unblinded subjective trial with positive results it's methodologically flawed worthless nonsense but if a psychologist conducts an unblinded subjective trial with positive results it's wonderful research that should be widely and uncritically reported.

the latter is common and normal practice not sure about the former.

 

[Woolie]

Thanks for this, @Sean, I found it really interesting!

The most relevant part for us is the bit about the researcher allegiance effect in psychotherapy. If you do a metanalysis of all psychotherapy studies meeting certain quality criteria, and also include as a predictor the particular allegiance of the researchers (which therapy they "believe in"), you get a massive effect: Cohen's d = 0.54, which is huge, p > .001.

In other words, you can predict with pretty good accuracy what the results of a psychotherapy study will be just by knowing what the researchers themselves believe in.

@Dolphin posted on this in an earlier thread:

http://forums.phoenixrising.me/inde...-overview-of-reviews-munder-et-al-2013.22466/

 

[chipmunk1]

what happens if you exclude that effect?

 

[A.B.]

what happens if you exclude that effect?

When rheumatologists (?) study CBT and GET for CFS, they find it's harmful (worse physical function and pain scores): Health-related quality of life in patients with chronic fatigue syndrome: group cognitive behavioural therapy and graded exercise versus usual treatment. A randomised controlled trial with 1 year of follow-up.

As for psychotherapy studies, I wonder what happens when researchers study a therapy they dislike / don't believe in.

 

[chipmunk1]

As for psychotherapy studies, I wonder what happens when researchers study a therapy they dislike / don't believe in.

In the sixties i believe behaviourists studied psychoanalysis and came to the conclusion that the more psychoanalysis a person had the less likely it was that the person would recover.

 

[Snow Leopard]

The lesson is: Be careful what you measure. What you think you are measuring might not be what you are actually measuring.