Scientific inbreeding and same-team replication: Type D personality as an example

[Dolphin]

This sounds interesting and potentially relevant for the ME/CFS field.

John Ioannidis is an interesting and influential researcher. Among other things, he wrote:

Ioannidis JP. Why most published research findings are false. PLoS Med. 2005 Aug;2(8):e124. Epub 2005 Aug 30

http://www.jpsychores.com/article/S0022-3999(12)00263-2/abstract

Journal of Psychosomatic Research

Article in Press
Scientific inbreeding and same-team replication: Type D personality as an
example

John P.A. Ioannidis

Received 11 September 2012; received in revised form 13 September 2012;
accepted 19 September 2012. published online 08 October 2012.

Abstract*

Replication is essential for validating correct results, sorting out false-positive early discoveries, and improving the accuracy and precision of estimated effects.

However, some types of seemingly successful replication may foster a spurious notion of increased credibility, if they are performed by the same team and propagate or extend the same errors made by the original discoveries.

Besides same-team replication, replication by other teams may also succumb to inbreeding, if it cannot fiercely maintain its independence.

These patterns include obedient replication and obliged replication.

I discuss these replication patterns in the context of associations and effects in the psychological sciences, drawing from the criticism of Coyne and de Voogd of the proposed association between type D personality and cardiovascular mortality and other empirical examples.

Keywords: Replication, Bias, Confirmation bias, Allegiance bias

* I gave each sentence its own paragraph

 

[alex3619]

The link is malformed, but all the info is there. The correct link appears to be:
http://www.jpsychores.com/article/S0022-3999(12)00263-2/abstract

This site has a paywall.

 

[Dolphin]

The link is malformed, but all the info is there. The correct link appears to be:
http://www.jpsychores.com/article/S0022-3999(12)00263-2/abstract

This site has a paywall.

Thanks. Fixed it so link should click through now (but, as you say, no extra info).

 

[alex3619]

I currently writing a blog on zombie science and another on information monopolies - both are connected to imbalances in information. So is this article. If research is all from similar sources, there is no research diversity, any failures in the study could appear in all the studies. Furthermore if peer reviews, editorial responses, media investigation, and the general scientific community do not hold a collection of papers up for much scrutiny, then this is essentially a monosource of information, even if its spread out in multiple papers.​

​

Diversity, criticism and alternative views are essential to science. Small insular research communities are at serious risk of getting so caught up in their own research that erroneous methodologies and conclusions can go unchallenged for very long periods of time.​

​

Similar arguments can be leveled at evidence based medicine.​

​

Bye, Alex​

 

[Esther12]

Thanks. Something else for my 'to read' list!

 

[biophile]

Classic example, the same group of CDC researchers "replicating" the childhood trauma association using the same questionable methodology.

 

[Esther12]

Researchers doing this sort of work probably are put off looking critically at CFS work because of the drama which surrounds the condition, which is a real shame, as they'd have plenty to get their teeth in to.

 

[Enid]

Ah something worth looking into - type A, B or C - full of good information no doubt.

 

[Dolphin]

In case people are wondering what obedient replication and obliged replication are:

In yet other cases, replications are done by different teams that are
nevertheless heavily influenced and not truly independent from the
original team and school of thought. This could include obedient replication,
where investigators feel that the prevailing school of thought
is so dominant that finding consistent results is perceived as a sign of
being a good scientist and there is no room for dissenting results and
objections; or obliged replication, where the proponents of the original
theory are so strong in shaping the literature and controlling the
publication venues that they can largely select and mold the results,
wording, and interpretation of studies eventually published. Truly
independent replication probably remains the exception in many scientific
fields.

I'm very interested in this whole topic and would be interested in reading some of the papers referenced.

 

[alex3619]

In case people are wondering what obedient replication and obliged replication are:
I'm very interested in this whole topic and would be interested in reading some of the papers referenced.

Doh, copyright infringement! (only joking). This is a part of where I am going with my book, and I have written up some early thoughts on this in one of my blogs:

http://forums.phoenixrising.me/index.php?entries/part-one-verificationism.1148/

In philosophy I would call this dogmatic verificationism. Its embracing old positivist modes of thought: its stuck in 19th century "science". Karl Popper called this "non-science". Its a failure to embrace the idea of critical analysis and testing. Superstition, the creation of self reinforcing ideas that are resistant to external influence, proliferates under these conditions.

I regard modern scientific consensus building, if not approached carefully, as a form of this problem - its "scientific" group think.

Bye, Alex

 

[jimells]

from "Why most published research findings are false"

claimed research findings may often be simply accurate measures of the prevailing bias.

At least something about these studies is accurate!

 

[Simon]

I think this is a striking paper, not only for what it says but also because it was published by the Journal of Psychosomatic Research, which has probably notched up a few of the psychosomatic research false positives referred to by Ioannidis and by Coyne.

Here's the abstract of the paper by Coyne & de Voogd appearing in the same issue as the Ioannidis paper:

Are we witnessing the decline effect in the Type D personality literature? What can be learned?
James C. Coyne, Jacob N. de Voogd, 2012

After an unbroken series of positive, but underpowered studies seemed to demonstrate Type D personality predicting mortality in cardiovascular disease patients, initial claims now appear at least exaggerated and probably false.

Larger studies with consistently null findings are accumulating. Conceptual, methodological, and statistical issues can be raised concerning the construction of Type D personality as a categorical variable, whether Type D is sufficiently distinct from other negative affect variables, and if it could be plausibly assumed to predict mortality independent of depressive symptoms and known biomedical factors, including disease severity. The existing literature concerning negative affect and health suggests a low likelihood of discovering a new negative affect variable that independently predicts mortality better than its many rivals.

The apparent decline effect in the Type D literature is discussed in terms of the need to reduce the persistence of false positive findings in the psychosomatic medicine literature, even while preserving a context allowing risk-taking and discovery. Recommendations include greater transparency concerning research design and analytic strategy; insistence on replication with larger samples before accepting “discoveries” from small samples; reduced confirmatory bias; and availability of all relevant data.

Such changes would take time to implement, face practical difficulties, and run counter to established practices. An interim solution is for readers to maintain a sense of pre-discovery probabilities, to be sensitized to the pervasiveness of the decline effect, and to be skeptical of claims based on findings reaching significance in small-scale studies that have not been independently replicated.
========

Actually, this seems quite polite compared with a letter they wrote to the editor critiquing a previous meta-analysis linking Type D personality to cardiac mortality. Coyne also published a large study in 2011 refuting the link between Type D personality and heart failure.

 

[Simon]

Thought it worth highlighting a few key points from the main Ioannidis paper:

First, he explicitly uses the problems seen in false Type D personality associations in research as examples of a wider malaise in psychological science (though don't forget he's already had a pop at most other forms of Life Science):

Many psychological sciences, including psychosomatic research, seem to have endorsed patterns of investigation that can sometimes favor scientific inbreeding, manifested by same-team replication, and potentially also obedient replication, or even obliged replication. In this issue, Coyne and de Voogd meticulously dissect the evidence surrounding the association of type D personality with cardiovascular mortality [1]. Their examination of the data and their shortcomings is exemplary.

Type D personality and cardiovascular mortality offer a very interesting case study that may not be better or worse than many other associations that have occupied many scientists in psychosomatic research and beyond. The features of this case study may apply to many other proposed associations and effects.

Later he suggests reasons why psychological studies might be partiuclarly prone to certain biases:

The psychological sciences may be particularly susceptible to these scenarios because many of the psychological variables and outcomes measured and analyzed are often convoluted, complex, and highly correlated and there is large flexibility in definitions, uses of cut-offs, modeling, and statistical handling of the data, hence large room for exploratory analyses and vibration of effects [vibration effects occur where small undeclared changes to the analysis eg excluding outliers or focusing on a sub-group can lead to dramatically different findings].

Thinking of the PACE trial, coming up with arbritary and flawed cut-offs (thresholds) for 'normal' would seem to be a perfect illustration even though it was done with non-psychological measures. Similarly, they published the percentage improving by a certain amount in the trial while failing to publish the percentage who got worse by the same amount (currently being challenged by a brilliant FOI request, see PACE thread). Again, this is an example of flexibility in analysis leading to overly positive findings.

And Ioannidis also highlights the lack of transparency, though points out this is an issue in many other scientific fields too:

it is common practice for researchers to keep their raw data, protocols, and analysis algorithms and codes close to their chest rather than sharing them in public.

Finally, he discusses how group-think and academic biases might play an important role in the development and perpetuation of false positive findings in psychological research.

...in fields that have high levels of inbreeding and one team has the lion's share of the major papers, it is likely that submitted papers will hit either one of the team members or an affiliate or devoted follower in the peer-review stage. For a paper to get published, it may have to pay its dues to the prevailing theory...

...Financial conflicts related to industry products may not be as common in the psychological sciences as in medicine, but academic conflicts related to allegiance and confirmation biases [30,31]may be as strong and influential in shaping the literature eventually

[says he thinks deliberate fraud is very rare but]
... conscious, subconscious and unconscious bias is probably very common and even intricately linked to the scientific discovery process. Independent replication is an essential tool to try to remove bias as much as possible and to understand true effects of associations and interventions.

As an antidote to such bias and group-think he proposes a wider range of researchers collaborating in the same study to give diversity of views:

The participation of many investigators and teams, which would not necessarily all endorse the same particular theory may offer some further safeguard of objectivity

Personally, I find the lack of thinking beyond a fairly rigid and unproven but widely accepted BPS model for CFS is one of the most depressing aspects of CFS research today.

 

[Simon]

I've just blogged about the Ioannidis and Coyne papers:

Fresh wind blowing through psychological research
though the title looks a bit inapproriate today with superstorm Sandy ripping through the US.

 

[Dolphin]

Here's the abstract of the paper by Coyne & de Voogd appearing in the same issue as the Ioannidis paper:

Are we witnessing the decline effect in the Type D personality literature? What can be learned?
James C. Coyne, Jacob N. de Voogd, 2012

I just came across something else Coyne wrote that I previously highlighted (contains link to the free text)

Are most positive findings in health psychology false.... or at least somewhat exaggerated?

James C. Coyne*1, 2

http://forums.phoenixrising.me/inde...e-or-at-least-somewhat-exaggerated-2009.19120