• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

Sense about science document on What is peer review

wdb

Senior Member
Messages
1,392
Location
London
The trouble with peer review is it's often a total circle-jerk, you have psychologists peer reviewing psychology papers, homeopaths reviewing homeopathy papers, etc, journals may even ask you to suggest suitable reviewers for your own submissions. Most crackpot researchers have enough like-minded crackpot peers to rubber-stamp all the research they like. I'd like to see it made much more independent, like non-peer review, papers are reviewed by researchers well outside of your circle who specialise sound methodology and analysis practices.


Also sense about science: http://senseaboutscience.org/activities/2012-john-maddox-prize/
 

alex3619

Senior Member
Messages
13,810
Location
Logan, Queensland, Australia
The trouble with peer review is it's often a total circle-jerk, you have psychologists peer reviewing psychology papers, homeopaths reviewing homeopathy papers, etc, journals may even ask you to suggest suitable reviewers for your own submissions. Most crackpot researchers have enough like-minded crackpot peers to rubber-stamp all the research they like. I'd like to see it made much more independent, like non-peer review, papers are reviewed by researchers well outside of your circle who specialise sound methodology and analysis practices.

I completely agree. There is a growing need for reviewers to be independent and well versed in sound methodology and analysis. Review circles may be even more damaging than citation circles.
 

Snowdrop

Rebel without a biscuit
Messages
2,933
Agree. In posting this I was hoping to point out that the establishment treats the idea of peer review as laid out akin to stone tablets from on high.

Clearly the peer review process is not up to standard or fit for purpose since it allows things like the PACE trial to be seen as gods gift to peer reviewed research.

There are some waves being made I think (around the edges) about peer review. Clearly needs a reformation.
 
Messages
2,158
Maybe the science journals need to employ experts at peer review to do the job full time, properly paid and given time to study the raw data and protocols properly. This would necessarily drastically cut the number of papers getting published. Which would be a good thing.

Maybe there could also be a different sort of online repository for any papers people want to put out, with attached raw data, data analysis, protocols etc inviting post publication open reviews.

And a separate open repository for the pilot studies and too small to be valid studies including most PhD papers to be summarised.

The whole academic sausage factory, publish or perish system seems to be extremely dysfunctional, especially in psychology / psychiatry where there seems to be little understanding of scientific method and a crisis of replicability.

Edit: I guess I should really read the Sense about science document before commenting...
 

Snowdrop

Rebel without a biscuit
Messages
2,933
Yes, I expect there are a number of ways to come at the problem and fix it.

I suspect though that any new protocols may attract their own problems too.

In this particular case I think looking outside to see if there is somewhere else that already deals well with the problem of peer review is a good starting place and then applying new measures on a trial basis one at a time in order to assess their effectiveness.
 

chipmunk1

Senior Member
Messages
765
Sense about Science seems more like a lobbying group with S. Wessely on board. Run by PR people.

The reason why they are promoting peer review as reliable is because the process is broke and unreliable in reality.

They want you to follow scientific publications like a religion. (Infallibility)

If the priests say something is true you must follow it. If you do not believe what Priest Review says you are irrational or you do not understand Science(God).

Not sure what the agenda is but they seem to be closely related to the Science Media Centre, a PR agency pretending to be scientists while making sure that industry tobacco science is getting the media exposure it deserves. A thought police for journalists.
“the SMC does not believe that ‘links’ to industry automatically equate to a compromise in the quality or integrity of a piece of science”.

Both of course against the view that ME is a real illness.
 
Last edited:
Messages
2,158
Edit: Sarcasm alert:

On their logic, both the PACE trial and the Caroline Wiltshire et al. recent paper demolishing it are published in peer reviewed journals, so are both true. I wonder how they rationalise that one.

Oh, yes, I remember, they get their chums to write rude and offensive peer reviews of her next paper...
So that's all right then. Nothing to see here folks, move along.
 
Last edited:

barbc56

Senior Member
Messages
3,657
Very interesting topic but certainly confusing as far as the intent of Sense about Science..The award for 2015 is Edzard Ernst, a debunker of pseudoscience. I think he's a credible scientist. However, I'm not sure what his stance is on me/cfs.

The winners seem to be "survivors of personal criticism", which is irritating and maybe even irrelevant when it comes to good science.

http://senseaboutscience.org/activities/2015-john-madd
 
Messages
2,158
Edit: Sarcasm alert!

Ah, yes, who could forget the award of the Maddox prize to that brave seeker after truth S Wessely. :vomit::vomit::vomit::vomit::vomit::vomit::vomit::vomit::vomit:

I'm just waiting for the day the equally heroic E Crawley :alien: gets nominated. Her recent talk seems to indicate that she just knows in her dear little beating heart that she deserves it.

Edit: Do I need to put a sarcasm alert on my posts? Probably not.

Later edit: Done, thanks for the reminder, @alex3619
 
Last edited:
Messages
2,391
Location
UK
If the journal’s editor thinks it is suitable for their journal they send the paper to other scientists who research and publish in the same field
This is where it can all go so horribly wrong ... right at the start in fact. If the journal is biased, then their choice of reviewers is going to be biased. And aside from that, if the reviewers are confined to others from the same field and who also publish, we are heading into the realms of heavy selection bias of reviewers.

I can understand good reasons why you must include reviewers from the same field - of course you must. But there are also many things common across science, and a good sprinkling of other stakeholders is usually crucial. In engineering peer reviews there will often be a wide spread of expertise amongst reviewers, to catch those issues the "usual experts" might miss.
 

RogerBlack

Senior Member
Messages
902
Open review can at least bring this into the open.
Especially open review where outsiders can comment on the reviews.
Otherwise (to be charitable) reviewers can make bald statements which are in fact incorrect, and pull the wool over editors who are not skilled in the field.