JaimeS
Senior Member
- Messages
- 3,408
- Location
- Silicon Valley, CA
False-Positive Psychology:
Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
by: Joseph P. Simmons, Leif D. Nelson, Uri Simonsohn
Note: this is a 2011 article, not a new one
http://journals.sagepub.com/doi/full/10.1177/0956797611417632
It's a fascinating one, and very applicable to what we've found...
Abstract:
In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.
Keywords methodology, motivated reasoning, publication, disclosure
____________________________________
I couldn't find this article posted anyplace; if there's an old thread on it, please let me know.
Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant
by: Joseph P. Simmons, Leif D. Nelson, Uri Simonsohn
Note: this is a 2011 article, not a new one
http://journals.sagepub.com/doi/full/10.1177/0956797611417632
It's a fascinating one, and very applicable to what we've found...
Abstract:
In this article, we accomplish two things. First, we show that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.
Keywords methodology, motivated reasoning, publication, disclosure
____________________________________
I couldn't find this article posted anyplace; if there's an old thread on it, please let me know.