• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

(just general) "What effect size would you expect?" (blogpost)

Dolphin

Senior Member
Messages
17,567
Last edited:

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK
It highlights how, perhaps surprisingly, effect sizes of bigger than d>0.1 will occur very frequently by chance, especially with smaller studies.
I was a bit disappointed by the piece - Simons fails to mention confidence limits at all, and most of those effect sizes >0.1 would not be significant ie still a null result. Makes me wonder about psychology professors and stats....

The kind of simulations he talks about - drawing random samples from two populations with the same mean - are very common and do show bigger effect sizes on smaller studies, but the confidence interval for most of the effect sizes include 0, ie non-significant (sorry, there are blogs etc on just this but can't remember any right now).

He also warns about replications being judged on just having the same direction of effect size (eg group A bigger than B in original in and replication) and this could be common - even if no true difference - with an effect size of 0.1 on replication. But even if the replicated effect size of 0.1, or 0.2, was significant (it wouldn't be, unless the study was huge) it would impress no one precisely because it is a trivial effect size. So he seems to be tackling a straw man argument here.
 

Dolphin

Senior Member
Messages
17,567
I was a bit disappointed by the piece - Simons fails to mention confidence limits at all, and most of those effect sizes >0.1 would not be significant ie still a null result. Makes me wonder about psychology professors and stats....

The kind of simulations he talks about - drawing random samples from two populations with the same mean - are very common and do show bigger effect sizes on smaller studies, but the confidence interval for most of the effect sizes include 0, ie non-significant (sorry, there are blogs etc on just this but can't remember any right now).

He also warns about replications being judged on just having the same direction of effect size (eg group A bigger than B in original in and replication) and this could be common - even if no true difference - with an effect size of 0.1 on replication. But even if the replicated effect size of 0.1, or 0.2, was significant (it wouldn't be, unless the study was huge) it would impress no one precisely because it is a trivial effect size. So he seems to be tackling a straw man argument here.
Thanks. I forgot to look out for the significance/confidence intervals point.

I came to it via this analysis of a paper:
http://asehelene.wordpress.com/2014...-glass-into-an-oddly-analyzed-clinical-paper/
where the authors of the paper mentioned effect sizes without highlighting importance of confidence intervals:
Now effect sizes are nice, of course. In this case they run from the middling to large, and also include a few negative ones (suggesting that things got worse). But, one must remember that with only 38 participants, effect-sizes tend to be inflated, as the handy chart in Dan Simon’s blog shows (simulations of effect size estimates where the true effect size is zero – you can do that when you simulate).

The table also shows confidence intervals. I take it that it is for the effect sizes. I looked up how you calculate confidence intervals for effect sizes to try to make sense of this, and you can do it of course. It is a bit trickier than just calculating confidence intervals for estimated means – involving non-central non-symmetric t-distributions, but it can be done, and evidently there are nice R-algorithms for it.

The confidence intervals are large, and all go from a number less than zero to above. That is, for every single effect size, the “no effect whatsoever” is still within the possible estimate. There are likely a couple of typos there also – two confidence intervals are identical. One starts at the same number as the estimated effect size. (It is not the only place where the copy editors and proof readers missed. Figure 5 lacks labels on the axes.)
 
Last edited: