http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD003200.pub4/full#CD003200-sec1-0013
I've got issues with a number of the replies. Some of the issues are very complex to analyse. So I'll just address this one for now...
Cochrane said:
Selective reporting (outcome bias)
The Cochrane Risk of Bias tool enables the review authors to be transparent about their judgments, but due to the subjective nature of the process it does not guarantee an indisputable consensus. You particularly mention the risk of bias in the PACE trial regarding not providing pre-specified outcomes however the trial did pre-specify the analysis of outcomes. The primary outcomes were the same as in the original protocol, although the scoring method of one was changed and the analysis of assessing efficacy also changed from the original protocol. These changes were made as part of the detailed statistical analysis plan (itself published in full), which had been promised in the original protocol. These changes were drawn up before the analysis commenced and before examining any outcome data. In other words they were pre-specified, so it is hard to understand how the changes contributed to any potential bias. The relevant paper also alerted readers to all these changes and gave the reasons for them. Overall, we don’t think that the issues you raise with regard to the risk of selective outcome bias are such as to suspect high risk of bias, but recognize that you may reach different conclusions than us.
My comments are in green.
Selective reporting (outcome bias)
The Cochrane Risk of Bias tool enables the review authors to be transparent about their judgments, but due to the subjective nature of the process it does not guarantee an indisputable consensus. You particularly mention the risk of bias in the PACE trial regarding not providing pre-specified outcomes however the trial did pre-specify the analysis of outcomes. [
< PACE absolutely did not pre-specify the analysis of outcomes. Nice spin.]
The primary outcomes were the same as in the original protocol, although the scoring method of one was changed and the analysis of assessing efficacy also changed from the original protocol. [
< Correct to say that the primary analysis of efficacy was changed, which is the central complaint. The primary outcome measures (questionnaires) were the same (but with altered scoring), but the primary endpoint analyses were all switched, which is crucial.]
These changes were made as part of the detailed statistical analysis plan (itself published in full), which had been promised in the original protocol. [
< The statistical analysis plan was published after the main 2011 paper.]
These changes were drawn up before the analysis commenced and before examining any outcome data. [
< In other words the changes were implemented after the trial commenced and after data was being collected in an open-label study. In other words "post-hoc".]
In other words they were pre-specified [
< Not!], so it is hard to understand how the changes contributed to any potential bias. [
< Really? Even if you try very hard?]
The relevant paper also alerted readers to all these changes and gave the reasons for them. [
< I'd have to re-read the full paper to be certain but I'm fairly convinced that the 2011 paper did not comment on all of the changes, if any, and definitely did not include a sensitivity analysis.]
Overall, we don’t think that the issues you raise with regard to the risk of selective outcome bias are such as to suspect high risk of bias [
< Quel Surprise!], but recognize that you may reach different conclusions than us.
