What constitutes reliable evidence?

[barbc56]

I'm confused, is this thread about Dr. Myhill's proof that her treatments are effective or what is considered reliable evidence. Should the the two threads be merged?

I have a lot of citations for evidence based medicine which I can post when able.But which thread do I put it under?

May I assume it's this one?

 

[barbc56]

I think that Dr Myhill (rightly or wrongly) believes that she has been witness to enough 'proof' to make this statement. So in her eyes she is sticking to proven statements

i think you are correct. That's what I find scary about her.

I think I just contradicted what I said in my above post. Each post is really a separate topic. I think?

 

[skipskip30]

What specific criticisms do you consider unjustified ?

You say that critics should stick to proven statements then do you also think that Myhill should stick to proven statements ?

I think I was very clear about what I meant actually.

People should criticise her based on things that she has actually done/said and not presuming things that are false such as her not telling patients the chances of her therapies working or not. You can disagree with her experimenting with patients but they are told that its all uncertain and a fair amount of trail and error is involved.

Im not actually her biggest fan and I wouldn't go back to see her again for multiple reasons but I just like to stick to things that are actually true when critiquing someone.

 

[markielock]

Concerning the topic of this thread, I think an apt description for it would be discussing the reliability of evidence using Sarah Myhill as an example case.

I have read a lot of Sarah Myhill's work: her books and her wiki. Although this information has broadened my perspective, one thing that has always stood out to me over the years was the lack of visible references to back up the advice she was advocating. I would love a list of references for each section on her wiki or a section at the end of her book. If I have somehow missed where she publishes her references, please let me know . Currently, this has led me to make the assumption she has formed these ideas from trial and error treating and talking to people with similar symptoms. Although, this advice may well be true (at least, for some patients) it still reads like assumptions without the rigour and resilience of the scientific method to back it up (I'm not saying she is not following the scientific method. However, it's not clear because it's uncertain what she bases her hypothesis on).

I personally fear this strategy of disseminating advice may get in the way of progress if it creates a false perception of making progress on understanding ME/CFS. It could create untried 'givens' that end up getting in the way of progress (whether that is in the academic setting, social issues and tolerance or more bad press that could hinder awareness and funding etc...) and getting to the core of the issue: what is ME/CFS and how do we treat it? We need to set a precedent or create a universal framework for ethically disseminating our theories and 'best advice we currently have' that is based on the scientific method and using a process of elimination. This is so people are not confused between theory or 'fact' (by fact I mean, our strongest, most credible theories). We have a responsibility as a community and as individuals, especially if we're in a position of influence, to be objective and transparent when it comes to understanding and communicating ME/CFS.

All I would ask to be done differently is if the theories and advice coming from Sarah Myhills domain are more clearly defined: whether they are experimental and based solely on experience and how the conclusions were drawn etc...

I think the definition of reliable evidence has been summed up nicely already. Concerning the campaign, I also agree that it may come across a bit too emotionally charged and may be misinterpreted.

 

[Laelia]

I think the definition of reliable evidence has been summed up nicely already.

Hello @markielock, please can you tell me where and how the definition of reliable evidence has been summed up already. Many thanks

 

[wdb]

Whatever strength of evidence we decide constitutes reliable evidence one thing we need to do is to apply that threshold more consistently.

There are many methodological flaws that produce bad misleadng evidence that often get pointed outed in CBT/GET trials such as:

• broad inclusion of ME/CFS/CF sufferers
• relying on subjective self reporting of effectiveness
• not controlling for placebo, suggestibility and practitioner influence
• cherry-picking what results to share or to emphasise
• downplaying non-responders or those who report worsening

These are bad practices whoever is engaging in them, it is absolutely right to point out bad research when Peter White or Esther Crawley does it and it is absolutely right to point it out when Dr Myhill does the same thing. Can we try and stop treating people as heroes who point out these flaws in research we don't like and villains when they point out exactly the same flaws in research we do like.

 

[markielock]

Hello @markielock, please can you tell me where and how the definition of reliable evidence has been summed up already. Many thanks

Hey @Laelia, Certainly . I personally felt @Alvin2 on page 1 and @alex3619 on page 2 summed up the concept of evidence and its reliability well. I found their answers expressed the nuances of what evidence is; that it is a spectrum of validity and isn't always black and white. However, it provides a clear audit trail and increases the chances of coming to logical conclusions.

Hope this helps clear up what I was referring to .

 

[Laelia]

Hey @Laelia, Certainly . I personally felt @Alvin2 on page 1 and @alex3619 on page 2 summed up the concept of evidence and its reliability well. I found their answers expressed the nuances of what evidence is; that it is a spectrum of validity and isn't always black and white. However, it provides a clear audit trail and increases the chances of coming to logical conclusions.

Hope this helps clear up what I was referring to .

Thank you very much @markielock for clarifying that

 

[Laelia]

Hello again @markielock

I would love a list of references for each section on her wiki

There are lots of references on her website (some pages appear to be better referenced than others).

or a section at the end of her book

She has reference section at the end of her CFS book too, 24 studies in all.

All I would ask to be done differently is if the theories and advice coming from Sarah Myhills domain are more clearly defined: whether they are experimental and based solely on experience and how the conclusions were drawn etc...

I'm a bit confused by this because it appears to me that Dr Myhill goes to great lengths to explain her theories and what studies she is using to back up her claims (hence producing a whole book about it!). I am not a professional scientist though so perhaps I am not a good judge... :thumbdown:

 

[markielock]

@Laelia, thanks for pointing out my mistake. You're right, she does have a references section at the back of the book! I even remember now that one of the reasons why I bought the book was that I saw it had a references section and I thought that may illuminate to the sources the advice may be based on in the wiki. You'll have to forgive my foggy forgetfulness!

I'm not here to argue for or against Sarah Myhill. I actually really quite like her and her tenacity. I don't think it will be necessary to offer any more opinions on how I feel about her referencing nor the studies she is referring to. I would rather just highlight my main point again regarding an agreed framework for everyone to consistently present and disseminate information where there is little room for interpretation of its origin and reliability.

 

[Laelia]

Hello Jonathan,

Another point. I wonder whether you might be mistakenly using the term 'reliable evidence' to refer to only evidence that has been proven to be reliable. This is perhaps where some of the confusion arising from. Surely evidence can sometimes be reliable even though it's not proven to be reliable?

So sorry Jonathan, please ignore this comment I made earlier. This appears to be another of my brain fog moments where I wasn't making any sense. I hope I will not make a habit of doing this. Apologies (again) for wasting your time.

@Jonathan Edwards

 

[Laelia]

Hello @Knockknock,

Thank you for your contribution to this thread

"" what large studies you guys are asking for?? What trials?? What science?? We all know that have been denied forever to me/cfs!!
Science is being denied blocked missleaded for us!!
They vey little we have its from theories based on personal experiences, doctors experiences, brave doctors and researchers that have treated their patients, that have done their research depite their limitations, lack of resourses and resistance and oposition from the govermments and their heath institutions.
We are not HIV/AIDS, we are not MS... we are ME/CFS.. the forgotten plague!!!

Exactly! Well said!

We all have to respect every one opinion on this threads, they are for everyone to express their opinion, but something I have noticed, there is a subset of pöeople( members) they Always run to challenge, to discredit, to disproved everything, from recovery stories of patients( me/suffers) to treatments that Doctors are using base on their long experiance with their patients, by seen how they respond to treatment.

I just want to point out that those who are doing the challenging are doing it with the very best of intentions. They are trying to protect us from what they see as bad science, in exactly the same way that they are trying to protect us from the bad science of the PACE trial.

 

[Laelia]

There seems to be widespread misunderstanding of what good evidence is. Evidence based medicine is about reducing risk of biased outcomes as much as possible. There are no guarantees. Indeed many controlled case studies are much better than many double blinded RCTs, and many dbRCTs are quite rubbish. RCTs are even worse. Its all about the details.

Hi Alex

Are you saying here that controlled case studies can sometimes provide us with reliable evidence?

I strongly believe some of these doctors are sidetracking any real progress and possible treatments for me/cfs

Hello Barb

Are you able to explain a bit more about how you feel some doctors are sidetracking progress into treatments for ME/CFS?

If Myhill has real evidence, if others have really improved from these "maverick doctors " point me to that evidence. Then I might think more highly of them

I will repeat what I said earlier in this thread:

There are many recovery stories on here with significant numbers reporting that their improvements were linked to the sorts of treatments Dr Myhill recommends. Of course we don't know whether all these people have correctly identified the reasons for their recovery and so it’s difficult to draw any conclusions from this. But the anecdotal evidence is there for sure.

Should the the two threads be merged

Which other thread are you referring to here @barbc56?

i think you are correct. That's what I find scary about her.

I don't find it scary, I find it interesting. I think it provides a good illustration of the way in which people have different interpretations of what reliable evidence is.

 

[alex3619]

Are you saying here that controlled case studies can sometimes provide us with reliable evidence?

Yes. It depends on the effect size. A huge effect is very obvious. Subtle effects will get missed.

 

[Laelia]

Yes. It depends on the effect size. A huge effect is very obvious. Subtle effects will get missed.

Thank you Alex. Aren't case studies based largely on anecdotal evidence? Does that mean that you think that anecdotes can sometimes provide reliable evidence?

 

[arewenearlythereyet]

I think this thread is going around in circles a bit.....some of this is Brain fog which is totally fair enough no judgement here! . Here is my simplistic view:

All scientific studies start with an idea which can come from any source...there is a bit of methodology you could apply to generate ideas but in the main this shouldn't be too restrictive. Once you have the idea though there is clearly laid out protocols and experimental design you can apply to eliminate bias (isolation of variables or more complex multi factorial experimental design). Once you have experimented and have enough evidence to isolate treatment options you can validate via clinical trials etc.

Then comes treatment which is regulated monitored and controlled.

Its as simple as that?

Some practitioners (normally private) bypass the validation stage because it is very expensive and time consuming for a small practice to carry out. Some of the more irresponsible advocate an abandonment of scientific process altogether and advocate a more "holistic" approach.

Dr Myhill is a funny mixture of part science and part holistic, but she doesn't appear to refer much to validated trials or research the subject field thoroughly enough to give a balanced view from what I can see. Patients entering into these treatments do so at their own risk and should always try and weigh up the risks of doing so. A good practitioner will lay these risks out for you before giving you treatment. Many GP's in the NHS do not do this. Sarah Myhill is an ex NHS GP so she doesn't have a great pedigree in research or scientific experimental design to speak with authority in this area. This is apparent by much of the vast content on her website.

If you are unsure, you simply have to ask the practitioner for validation trial information. If this doesn't exist its more risky than not since there is no proof that it actually works or doesn't give harmful side effects.

However there is always risk regardless of trial. This is not black and white and will never be so. This is why all pieces of research have to be weighed up on their own merits. The more pieces of research by different researchers saying the same thing gives a level of confidence.

Whether you try any treatment is up to you and its always a balance of risk vs benefits. Your attitude to risk has a bearing on whether to try or not. We will all differ in this respect.

 

[markielock]

Thanks @arewenearlythereyet , you articulated most of what I was struggling to vocalise

 

[Laelia]

I think this thread is going around in circles a bit.....some of this is Brain fog which is totally fair enough no judgement here! .

Yes there is definitely an issue with brain fog!

I don't think it's just about the brain fog however, I think there might be a fundamental disagreement about what constitutes reliable evidence.

Some practitioners (normally private) bypass the validation stage because it is very expensive and time consuming for a small practice to carry out. Some of the more irresponsible advocate an abandonment of scientific process altogether and advocate a more "holistic" approach.

But is it an abandonment of the scientific process? Or is it just a case of not having the resources and funds to put their hypotheses through expensive clinical trials? I know some say that people should not sell treatments if they have not been through the validation process and I understand that there is a very good argument behind this. However I believe that this type of 'experimental' medicine is working for a lot of people so for that reason I think I am in favour of it (providing that this is made clear and the risks explained).

Sarah Myhill is an ex NHS GP so she doesn't have a great pedigree in research or scientific experimental design to speak with authority in this area. This is apparent by much of the vast content on her website.

Yes I appreciate that Sarah Myhill is not a specialist in research and experimental design. However I suspect that she may have some strengths in other areas that those in research might be missing. I won't go into that though as this is not the place for it.

 

[arewenearlythereyet]

But is it an abandonment of the scientific process?Or is it just a case of not having the resources and funds to put their hypotheses through expensiveclinical trials?

It depends. If its an idea that has had no experimental testing or validation before treatment, then "the treatment" could be classed as the experiment and therefore "experimental". Putting aside the ethics of experimenting on humans and all that entails, you could then argue that providing there is a well applied scientific rigour to the design this would still be scientific. If it's just a GP prescribing some stuff and hoping for the best without writing down results in a controlled way then no, this is not science at all.

Obviously one outcome of the treatment if applied scientifically could be that a number of the patients get worse or die due to side effects of whatever treatment is given. This is useful data for the scientific process providing it's designed well and meaningful but not so great for the patient. That is why we sometimes experiment first on the bench then on animals before testing on humans. It is dependent upon the risk assessment for the proposed experiment. For a good risk assessment and appropriate experimental design, you need to be an expert in your field and trained in designing experiments. Many of the quacks on the Internet are neither. Just having passed some exams once and having experience of prescribing treatments as a GP or whatever doesn't mean you are a scientist. I don't view GPs as the science end of medicine...the pharmacist is more scientific than the GP.

As far as poor doctors not having enough money to do clinical trials .....I would say if you don't have the cash to splash on a clinical trial, then perhaps don't offer advice or treatments that could be misconstrued as being validated. Always put a disclaimer for what you say? I personally think the medical field needs a kick up,the proverbial with regards to advising their clients of risk. Perhaps they should take a look at some other industries and bring themselves up to date a bit? You get more regulation and advisement of risk in the food, automobile, and finance sectors in this respect. Clients also have more recourse in these fields when things go wrong.

As far as being a human Guinea pig, this isn't something I would want for myself because I have seen what doctors can do by prescribing inappropriate treatment when they don't know what they are doing. If you are carefree and not risk averse then you could convince yourself that going down this road is something of benefit, but I would always advise to weigh up risks.

 

[Laelia]

That is a surprisingly good study it's on sci-hub is anyone wants to read it. They did a good job of blinding and controlling for expectation and influence and the results look quite strong. Only weakness I see are that it was quite a small sample (15 in treatment and 17 controls) and only lasted 6 weeks so it's unknown if the effect could be maintained. Also the Nottingham Health Profile looks like a bit of a blunt instrument.

So is this study a reliable or an unreliable piece of evidence?