"How peer reviewers might hold the key to making science more transparent" - Guardian

[sarah darwins]

This article (published Friday 16th Jan 2016) has a general focus on questions of transparency and data sharing in science, and touches on issues of interest to us:

In many cases, there are justifiable reasons as to why it would be inappropriate to publish data – for instance, if the study is about a very specific set of people with a rare medical disorder, it may still be possible to identify individuals from anonymised data. But regardless of whether or not the materials and data are publicly available, there should always be a clear justification as to why (or why not) within the paper itself. As long as reviewers are happy with the justification, the review process can go on as usual.

Link to full article: http://www.theguardian.com/science/...ld-the-key-to-making-science-more-transparent

The author is Pete Etchells — "the Guardian's science blog network coordinator". He has coauthored a paper suggesting a new approach:

But what about bottom-up approaches to the problem of promoting open science?

On Wednesday, a new paper published in Royal Society Open Science argued for a new, grassroots approach to this problem, by putting the power back into the hands of scientists at the coalface of research, by changing the way that we think about the peer review process (full disclosure: both myself and fellow Head Quarters blogger Chris Chambers are co-authors on the paper).

Only a dozen or so comments, all apparently from people working in research and nearly all against the proposal. I was struck by this phrase in one comment — "And once again I see Chambers and Etchells seeing all scientific endeavours as though they are flaky psychology projects which isn't so." Do most scientists still see psychology as flaky non-science? In which case, why aren't they more vocal about it? Isn't it harming their reputations, too?

 

[Sasha]

I added a comment about PACE - please feel free to upvote it so it stays high!

http://www.theguardian.com/science/...ing-science-more-transparent#comment-66854393

 

[Jonathan Edwards]

I think most scientists do see clinical psychology as flaky. I don't really get a feel for what this author is trying to tackle. There is an increasing problem with people not showing raw data and using things like box plots and I agree that should be reversed. But the main effect for me is to assume studies that do not show raw data are probably not very good so I put less weight on them. I guess the point is that in psychological studies it becomes an awful lot more important - as we all know.

 

[Sasha]

I think most scientists do see clinical psychology as flaky. I don't really get a feel for what this author is trying to tackle. There is an increasing problem with people not showing raw data and using things like box plots and I agree that should be reversed. But the main effect for me is to assume studies that do not show raw data are probably not very good so I put less weight on them. I guess the point is that in psychological studies it becomes an awful lot more important - as we all know.

I don't follow you - why is it more important in psychological studies? Surely it's important in any study that matters (including pharmacology studies in medical trials)?

Although you might (reasonably) assume that studies that don't show raw data are possibly weak, those aren't the sort of considerations that go into systematic reviews, which influence NICE guidelines. I suspect that most clinicians (GPs, at least) are probably not going to look at individual studies, and will never be aware of which don't share their data and should therefore be trusted less.

 

[sarah darwins]

I would think releasing data is even more important for psych studies because of the pitifully low replicability. You can’t even try to replicate a lot of psych studies without access to the exact same materials used by the authors. Whereas if, say, a biochemist claims that certain tissue cells die in vitro when exposed to substance X, any half-competent biochemist can put together an experiment to test that claim without much more information than that. Claims by psychologists tend to be awfully hard to verify or disprove. As we know (understatement).

 

[Bob]

The Peer Reviewers’ Openness (PRO) Initiative is, at its core, a simple pledge: scientists who sign up to the initiative agree that, from January 1 2017, will not offer to comprehensively review, or recommend the publication of, any scientific research papers for which the data, materials and analysis code are not publicly available, or for which there is no clear reason as to why these things are not available. To date, over 200 scientists have signed the pledge.

James Coyne signed up to the PRO initiative...

https://twitter.com/i/web/status/687996939355422720

And this week he posted this related blog:
What reviewers can do to improve the trustworthiness of the psychotherapy literature.
http://forums.phoenixrising.me/inde...worthiness-of-psychotherapy-literature.42461/

 

[worldbackwards]

I believe that this comment is the final word on data sharing:

 

[adreno]

Well, data can cost a fortune to collect. Imaging data is notoriously difficult and expensive to aquire. It's not like filling out a questionnaire. I suppose this is a problem, especially when institutions are competing. I can see why he is reluctant to share it.