Sense about science document on What is peer review

wdb

Senior Member
Messages
1,390
Likes
2,970
Location
London
The trouble with peer review is it's often a total circle-jerk, you have psychologists peer reviewing psychology papers, homeopaths reviewing homeopathy papers, etc, journals may even ask you to suggest suitable reviewers for your own submissions. Most crackpot researchers have enough like-minded crackpot peers to rubber-stamp all the research they like. I'd like to see it made much more independent, like non-peer review, papers are reviewed by researchers well outside of your circle who specialise sound methodology and analysis practices.


Also sense about science: http://senseaboutscience.org/activities/2012-john-maddox-prize/
 

alex3619

Senior Member
Messages
13,795
Likes
37,490
Location
Logan, Queensland, Australia
The trouble with peer review is it's often a total circle-jerk, you have psychologists peer reviewing psychology papers, homeopaths reviewing homeopathy papers, etc, journals may even ask you to suggest suitable reviewers for your own submissions. Most crackpot researchers have enough like-minded crackpot peers to rubber-stamp all the research they like. I'd like to see it made much more independent, like non-peer review, papers are reviewed by researchers well outside of your circle who specialise sound methodology and analysis practices.
I completely agree. There is a growing need for reviewers to be independent and well versed in sound methodology and analysis. Review circles may be even more damaging than citation circles.
 

Snowdrop

Rebel without a biscuit
Messages
2,933
Likes
10,161
Agree. In posting this I was hoping to point out that the establishment treats the idea of peer review as laid out akin to stone tablets from on high.

Clearly the peer review process is not up to standard or fit for purpose since it allows things like the PACE trial to be seen as gods gift to peer reviewed research.

There are some waves being made I think (around the edges) about peer review. Clearly needs a reformation.
 

trishrhymes

Senior Member
Messages
2,157
Likes
17,709
Maybe the science journals need to employ experts at peer review to do the job full time, properly paid and given time to study the raw data and protocols properly. This would necessarily drastically cut the number of papers getting published. Which would be a good thing.

Maybe there could also be a different sort of online repository for any papers people want to put out, with attached raw data, data analysis, protocols etc inviting post publication open reviews.

And a separate open repository for the pilot studies and too small to be valid studies including most PhD papers to be summarised.

The whole academic sausage factory, publish or perish system seems to be extremely dysfunctional, especially in psychology / psychiatry where there seems to be little understanding of scientific method and a crisis of replicability.

Edit: I guess I should really read the Sense about science document before commenting...
 

Snowdrop

Rebel without a biscuit
Messages
2,933
Likes
10,161
Yes, I expect there are a number of ways to come at the problem and fix it.

I suspect though that any new protocols may attract their own problems too.

In this particular case I think looking outside to see if there is somewhere else that already deals well with the problem of peer review is a good starting place and then applying new measures on a trial basis one at a time in order to assess their effectiveness.
 

chipmunk1

Senior Member
Messages
765
Likes
2,814
Sense about Science seems more like a lobbying group with S. Wessely on board. Run by PR people.

The reason why they are promoting peer review as reliable is because the process is broke and unreliable in reality.

They want you to follow scientific publications like a religion. (Infallibility)

If the priests say something is true you must follow it. If you do not believe what Priest Review says you are irrational or you do not understand Science(God).

Not sure what the agenda is but they seem to be closely related to the Science Media Centre, a PR agency pretending to be scientists while making sure that industry tobacco science is getting the media exposure it deserves. A thought police for journalists.
“the SMC does not believe that ‘links’ to industry automatically equate to a compromise in the quality or integrity of a piece of science”.
Both of course against the view that ME is a real illness.
 
Last edited:

trishrhymes

Senior Member
Messages
2,157
Likes
17,709
Edit: Sarcasm alert:

On their logic, both the PACE trial and the Caroline Wiltshire et al. recent paper demolishing it are published in peer reviewed journals, so are both true. I wonder how they rationalise that one.

Oh, yes, I remember, they get their chums to write rude and offensive peer reviews of her next paper...
So that's all right then. Nothing to see here folks, move along.
 
Last edited:

barbc56

Senior Member
Messages
3,656
Likes
4,953
Very interesting topic but certainly confusing as far as the intent of Sense about Science..The award for 2015 is Edzard Ernst, a debunker of pseudoscience. I think he's a credible scientist. However, I'm not sure what his stance is on me/cfs.

The winners seem to be "survivors of personal criticism", which is irritating and maybe even irrelevant when it comes to good science.

http://senseaboutscience.org/activities/2015-john-madd
 

trishrhymes

Senior Member
Messages
2,157
Likes
17,709
Edit: Sarcasm alert!

Ah, yes, who could forget the award of the Maddox prize to that brave seeker after truth S Wessely. :vomit::vomit::vomit::vomit::vomit::vomit::vomit::vomit::vomit:

I'm just waiting for the day the equally heroic E Crawley :alien: gets nominated. Her recent talk seems to indicate that she just knows in her dear little beating heart that she deserves it.

Edit: Do I need to put a sarcasm alert on my posts? Probably not.

Later edit: Done, thanks for the reminder, @alex3619
 
Last edited:

Barry53

Senior Member
Messages
2,391
Likes
13,643
Location
UK
If the journal’s editor thinks it is suitable for their journal they send the paper to other scientists who research and publish in the same field
This is where it can all go so horribly wrong ... right at the start in fact. If the journal is biased, then their choice of reviewers is going to be biased. And aside from that, if the reviewers are confined to others from the same field and who also publish, we are heading into the realms of heavy selection bias of reviewers.

I can understand good reasons why you must include reviewers from the same field - of course you must. But there are also many things common across science, and a good sprinkling of other stakeholders is usually crucial. In engineering peer reviews there will often be a wide spread of expertise amongst reviewers, to catch those issues the "usual experts" might miss.
 

RogerBlack

Senior Member
Messages
898
Likes
2,905
Open review can at least bring this into the open.
Especially open review where outsiders can comment on the reviews.
Otherwise (to be charitable) reviewers can make bald statements which are in fact incorrect, and pull the wool over editors who are not skilled in the field.
 

user9876

Senior Member
Messages
4,556
Likes
18,064
Open review can at least bring this into the open.
Especially open review where outsiders can comment on the reviews.
Otherwise (to be charitable) reviewers can make bald statements which are in fact incorrect, and pull the wool over editors who are not skilled in the field.
Also reviewers can give their friends and fellow believers an easy time.