(just general) "What effect size would you expect?" (blogpost)

[Dolphin]

(This may (?) be a bit complicated for somebody who has never taken a statistics course).

I came across the following blog post (it was linked to by another blog):

What effect size would you expect?
Posted by Daniel Simons
http://blog.dansimons.com/2013/03/what-effect-size-would-you-expect.html

It highlights how, perhaps surprisingly, effect sizes of bigger than d>0.1 will occur very frequently by chance, especially with smaller studies.

 

[alex3619]

This is made worse by the issue that negative studies often don't get published, or get published in low impact journals.

 

[Simon]

It highlights how, perhaps surprisingly, effect sizes of bigger than d>0.1 will occur very frequently by chance, especially with smaller studies.

I was a bit disappointed by the piece - Simons fails to mention confidence limits at all, and most of those effect sizes >0.1 would not be significant ie still a null result. Makes me wonder about psychology professors and stats....

The kind of simulations he talks about - drawing random samples from two populations with the same mean - are very common and do show bigger effect sizes on smaller studies, but the confidence interval for most of the effect sizes include 0, ie non-significant (sorry, there are blogs etc on just this but can't remember any right now).

He also warns about replications being judged on just having the same direction of effect size (eg group A bigger than B in original in and replication) and this could be common - even if no true difference - with an effect size of 0.1 on replication. But even if the replicated effect size of 0.1, or 0.2, was significant (it wouldn't be, unless the study was huge) it would impress no one precisely because it is a trivial effect size. So he seems to be tackling a straw man argument here.

 

[Dolphin]

I was a bit disappointed by the piece - Simons fails to mention confidence limits at all, and most of those effect sizes >0.1 would not be significant ie still a null result. Makes me wonder about psychology professors and stats....

The kind of simulations he talks about - drawing random samples from two populations with the same mean - are very common and do show bigger effect sizes on smaller studies, but the confidence interval for most of the effect sizes include 0, ie non-significant (sorry, there are blogs etc on just this but can't remember any right now).

He also warns about replications being judged on just having the same direction of effect size (eg group A bigger than B in original in and replication) and this could be common - even if no true difference - with an effect size of 0.1 on replication. But even if the replicated effect size of 0.1, or 0.2, was significant (it wouldn't be, unless the study was huge) it would impress no one precisely because it is a trivial effect size. So he seems to be tackling a straw man argument here.

Thanks. I forgot to look out for the significance/confidence intervals point.

I came to it via this analysis of a paper:
http://asehelene.wordpress.com/2014...-glass-into-an-oddly-analyzed-clinical-paper/
where the authors of the paper mentioned effect sizes without highlighting importance of confidence intervals:

Now effect sizes are nice, of course. In this case they run from the middling to large, and also include a few negative ones (suggesting that things got worse). But, one must remember that with only 38 participants, effect-sizes tend to be inflated, as the handy chart in Dan Simon’s blog shows (simulations of effect size estimates where the true effect size is zero – you can do that when you simulate).

The table also shows confidence intervals. I take it that it is for the effect sizes. I looked up how you calculate confidence intervals for effect sizes to try to make sense of this, and you can do it of course. It is a bit trickier than just calculating confidence intervals for estimated means – involving non-central non-symmetric t-distributions, but it can be done, and evidently there are nice R-algorithms for it.

The confidence intervals are large, and all go from a number less than zero to above. That is, for every single effect size, the “no effect whatsoever” is still within the possible estimate. There are likely a couple of typos there also – two confidence intervals are identical. One starts at the same number as the estimated effect size. (It is not the only place where the copy editors and proof readers missed. Figure 5 lacks labels on the axes.)