I think there are a huge number of factors playing into this. Some scientific, some cultural, some just poor reasoning.
First the science:
1. The brain is so very hard to study. The complexity is extreme, and the tools cannot handle that. Its about the hardest thing we have ever studied in science.
2. The issues cross specialties. Psychiatrists, for example, don't have a sufficient skill set. It takes a team, and a team across disciplines.
3. Because of the difficulty with this research, psychiatry has accepted low grade methodology. For example, if using standards in biomedicine I do not think ANY diagnosis in DSM would survive. The standards are very very low. A lot of psychiatry uses and tolerates very poor study methodology, as exemplified by the PACE trial. They use scientific trappings and claims, and in my view this makes much of it pseudoscience at best.
4. Evidence based approaches fail here because of three main reasons:
A. Evidence based on what diagnosis? If the diagnostic criteria are highly flawed, with poor diagnostic accuracy, then what can be said about studies based on them?
B. There are biases in a lot of psychiatric research, that act to increase the probability of making a positive finding. PACE is loaded with them. When you do a review or meta-analysis using EB methods then you risk reinforcing results due to bias. EBM is designed to minimize biases, that is the goal, but if you allow poor methodology, with entrenched biases, then you risk rubber stamping poor research.
C. Vested interests can distort the research base, and this is more worrying when this is due to funding bias and political support, which ties in with Zombie Science. If there are only 10 RCTs, and they were all funded by a small group of interests, then simply doing a review or meta-analysis will fail due to funding bias. Its a huge problem. Again, methodology counts.
I am not anti-psychiatry. I am indeed anti-bad-psychiatry. Psychiatry could be greatly improved, but as has already been posted on this thread that will not happen fast unless they acknowledge and keep working on the issues.
Cultural factors:
1. Psychiatry has failed to embrace science. They kind of do science, and hence its pseudoscience. We NEED something like psychiatry, its a travesty for poor research standards to be accepted.
2. Science used to proceed on the philosophy of logical positivism. Keep supporting your claims, its up to others to prove you are wrong. Most of science moved on from this in the middle of last century, instead moving to critical rationalism. (For the record, I am a pancritical rationalist, which is related to this.) Critical rationalism basically says that ideas need to be tested, and the better the tests the more sure you can be of the results, but you can never be certain. So methodology is critical, again.
3. Doctors do not, as a generalization, complain nearly enough about poor research. They tolerate it. I think this happens in psych a lot due to lack of confidence, and fear of failure. If they complain of other's poor research methodology, what then of their own methodology?
4. Medical culture suppresses dissent. There is this notion of not doing harm to the profession. There is a long history of ignoring or covering up mistakes.
5. Doctors are given privileged status under law. That has to change. Its abused for too often. Just look at the patients with ME or CFS who have been sectioned against their will. Look at the outcomes.
Just to be clear, in case anyone is thinking I am anti-doctor, my heroes are doctors. Top of that list is Barry Marshall, who almost lost his medical licence because he was a "quack". He later shared the Nobel prize. Most doctors who make my list have either had additional education, or worked hard to expand beyond traditional medical roles. Many are researchers.
Low standards of reasoning:
1. Much of medical reasoning is heuristic, designed for fast approximate answers. Time is often critical, either due to a medical crisis or due to insurance or bureaucratic demands. Such reasoning is fast, but its subject to serious flaws. Those flaws are often difficult to see. This kind of thing has been most investigated in economics, but I think it applies in many disciplines.
2. Doctors are not adequately trained in logical reasoning. They fail to see fallacies. Medicine has heavily embraced the dogma and fallacy known as Appeal to Authority. The PACE trial, especially when you consider not just the papers but press releases and interviews, is a morass of so many layered fallacies that I wanted to write a book on it. Maybe my health will improve and I will finish one day.
3. Most doctors do not adequately understand statistics. According to Gigerenzer that is about 80%+ in the USA.
4. Many doctors do not understand the reasoning behind EBM. RCTs are not actually the gold standard, you have to add a string of caveats. Anecdotal evidence is still evidence. Etcetera. Understanding why certain things are done in EBM tells you how the rules should be applied, and also when they do not apply.
5. As a special case of my comment on EBM, bureaucratic and insurance authorities misapply evidence based findings. Those findings are generalizations, and they limit the options of both doctors and patients if applied dogmatically. Even if some guideline (and they call them guidelines but sometimes treat them as hard rules) is 95% useful, then that still means that one in twenty patients will be poorly served by them. Doctors need the flexibility, training and resources to adapt to patients who do not fit the neat pigeonholes. We see that in the ME and CFS community a lot. Doctors are being trained to be inflexible. The bureaucrats want inflexible doctors.
I may no longer be capable of writing my book right now, but I can still give you some pieces of my analysis along the way.