Invest in ME Conference 12: First Class in Every Way
OverTheHills wraps up our series of articles on this year's 12th Invest in ME International Conference (IIMEC12) in London with some reflections on her experience as a patient attending the conference for the first time.
Discuss the article on the Forums.

Psychology Journal Bans Significance Testing

Discussion in 'Other Health News and Research' started by Dolphin, Feb 27, 2015.

  1. Dolphin

    Dolphin Senior Member

    Messages:
    10,574
    Likes:
    27,644
    Woolie, barbc56 and Denise like this.
  2. Dolphin

    Dolphin Senior Member

    Messages:
    10,574
    Likes:
    27,644
    I was aware how unreliable p-values might be

     
    anciendaze, Woolie, barbc56 and 5 others like this.
  3. CBS

    CBS Senior Member

    Messages:
    1,502
    Likes:
    845
    Yes, p-values are widely mis-used. They are only a part of the story. Surprised that BASP is taking this stance. I've always referred to p-values as a measure of statistical stability, not significance. Significance is a much more complex issue but this seems in part to be an issue of editorial oversight, a system that values quantity over quality and a larger ignorance regarding basic scientific principles.
     
  4. Valentijn

    Valentijn The Diabolic Logic

    Messages:
    14,117
    Likes:
    44,097
    I'm not sure the p-value deserves to be banned - but I certainly trust it a lot more when it's accompanied by a nice scatterplot graph or very detailed values.
     
    biophile and Dolphin like this.
  5. Woolie

    Woolie Senior Member

    Messages:
    1,771
    Likes:
    13,180
    I don't want to offend anyone in the field, but this journal BASP doesn't seem to be one of the highest impact in its domain - perhaps they hope to create a bit of a stir to improve their profile?
     
  6. Woolie

    Woolie Senior Member

    Messages:
    1,771
    Likes:
    13,180
    Nice explanation, @Dolphin!
     
  7. anciendaze

    anciendaze Senior Member

    Messages:
    1,774
    Likes:
    4,502
    Researchers tend to follow procedures while ignoring logical assumptions on which their validity is based. The fundamental assumption in all of these techniques tends to be that you are dealing with normal distributions, but very few researchers go to any lengths to validate this. They will even use standard procedures in cases where there is strong evidence the underlying distribution is far from normal.

    A normal (Gaussian) distribution is one which is completely specified by two parameters: mean and variance, or standard deviation. All other moments are assumed to be zero or negligible. The Central Limit Theorem gives some conditions under which random processes with other distributions will combine produce normal distributions. Among the preconditions are additive combination, independence, the existence of means and bounded variance in these other distributions. It is possible for each of these conditions to be violated in practice.

    What it guarantees, in those cases where it applies, is that the distribution of means and variance of sample sets will be normal in the limit where you repeat sampling many times. This is far from guaranteeing that any particular small set of samples will form a normal distribution. Even if the sampling process does produce a normal distribution you still have the problem of estimating those two parameters. Mean values tend to be more reliable than standard deviation, yet the estimates of standard deviation have enormous effect on measures of significance. The computer experiments above illustrate very well how sensitive results are to those estimates, yet these are experiments done in a mathematical world where the underlying assumptions are, as far as possible, true by construction!

    In the situation where one of the processes being sampled has a Lévy distribution you cannot even count on the existence of a bounded standard deviation in the pure mathematical case. Any such computed standard deviation will be an artifact determined by the bounds set on sampling and the number of samples taken. Unfortunately for convenience, Lévy distributions are quite common in many fields. They should be suspected any time you are dealing with asymmetrical distributions which exhibit a pronounced extended tail in one direction.

    This is precisely what I saw in looking at Chalder's data on physical performance in the general population. I could even plausibly explain this as the result of a number of pathological processes combining multiplicatively to reduce performance. In the general population there was no exclusion of multiple pathological processes, and in the CFS subset the existence of any undetected pathologies was simply dismissed.

    You can even find examples in the literature where researchers who were warned that they were not dealing with normal distributions persisted, saying "sometimes you have to work hard to get a normal distribution". Such people are completely oblivious to the obvious contradiction between the idea of necessary independence during sampling and "working real hard" to make the resulting sample set look normal. If the resulting groupings all exhibit increased variance as a function of time, and the bounds are set in a way to exploit this, this will guarantee that a study will be able to report "statistically significant results" because of procedures based on invalid assumptions. Having a number of sample points right up against one bound is a great way to do this, even if you don't adjust bounds during the study.

    Even that stops short of adjusting bounds for entry to a trial without changing thresholds for recovery. Any field which allows such travesties to persist must be considered fundamentally suspect.
     

See more popular forum discussions.

Share This Page