The 12th Invest in ME Research Conference June, 2017, Part 2
MEMum presents the second article in a series of three about the recent 12th Invest In ME International Conference (IIMEC12) in London.
Discuss the article on the Forums.

What constitutes reliable evidence?

Discussion in 'Other Health News and Research' started by Laelia, Apr 14, 2017.

  1. Laelia

    Laelia Senior Member

    Messages:
    243
    Likes:
    444
    UK
    This question cropped up while some of us were debating the merits of Dr Myhill's campaign for a Public Inquiry into the Medical Abuse of ME Sufferers over on this thread:

    http://forums.phoenixrising.me/inde...care-and-treatment-for-people-with-m-e.49345/

    The discussion about what constitutes reliable evidence begins on page 3 of this thread and continues through to the end of page 5. If anyone wants to continue that discussion they can do so here.

    (Please note that all the quotes that I copied with my comments below have come from the above thread)
     
    Last edited: Apr 16, 2017
  2. Alvin2

    Alvin2 Senior Member

    Messages:
    880
    Likes:
    2,139
    I have not read that thread (and don't have the brain power right now) but reliable evidence is from scientific study that is done honestly, repeatably, without bias, with wisdom, common sense and further research indicated.
    Since i assume we are talking about ME/CFS, it depends on the subject matter, a population study with a survey of symptoms is about good questionnaires (not concrete but rigid yet fluid enough to be accurate) and data analysis (and openness) vs a metabolomics study which depends on a representative sample, data collection standards, reputability and strong peer review (not a perfect process but when done properly can work, when done as a farce is completely flawed). All studies should have their data released for reanalysis by other experts, though anonymization is usually an excellent idea.

    An excellent piece of evidence is one that determines the cause for the measured abnormalities, though often we do not yet have the technology to do this, but it should be talked about (often scientists lose sight of the big picture) and hopefully a study is a step towards developing that technology by providing data, direction and theories.

    Repeatability is perhaps the most important, underestimated and ignored facet of the scientific process
     
    markielock and Laelia like this.
  3. Laelia

    Laelia Senior Member

    Messages:
    243
    Likes:
    444
    UK
    Hi @MEMum. I’m just responding to your comment from the other thread: “I don't think we can expect Dr Myhill to do double-blind placebo controlled etc studies, given that all her patients are paying.However I feel that if significant numbers were getting well enough to return to work and get off benefits, then more of the ME community (PwME and carers) would have heard about it.Any comments I remember seem to have been of small, incremental changes.” [My bold]

    There are many recovery stories on here with significant numbers reporting that their improvements were linked to the sorts of treatments Dr Myhill recommends. Of course we don't know whether these people have correctly identified the reasons for their recovery and so it’s difficult to draw any conclusions from this. But the anecdotal evidence is there for sure.

    @Jonathan Edwards: “But if Dr Myhill has not done a proper trial she was presumably handing out treatments when she has no idea if they work or not.”

    I think Dr Myhill believes (rightly or wrongly) that the treatments she hands out do work. Yes, she does experiment when she is treating patients. I don't see a problem with this, providing it is made clear and the risks explained.

    @Jonathan Edwards: “I very much doubt she said to patients 'I have no idea if this works or not but try it because I have a theory it will work that I will never actually know the validity of because I am never going to test it properly'.”

    Actually she does admit that she sometimes tries treatments when she doesn't know if they will work or not. For example this is what she says on her website when explaining the indications for use of Enzyme Potentiated Desensitisation (EPD):

    “I do sometimes use EPD for the worst possible reason, that is I can't think of anything else to do when all else has been tried. However it is surprising how often this works!”

    (Note also here that Dr Myhill believes she is able to determine that this treatment works just from the clinical response).

    Many ME/CFS patients want to experiment with different treatments even if there is only poor evidence for their efficacy. For many of us, we fear that the risk of not trying anything is greater than the risk of experimenting. We need practitioners like Dr Myhill who can assist us through this process.

    @TiredSam: “The amount of evidence needed to demonstrate that what Dr Myhill is saying is nonsense is zero, so there's enough. If she provides no evidence for her claims then she hasn't given anyone any reason to believe them or to go to the effort of disproving them, there's nothing to disprove.” [My bold]

    It’s not the case that she provides no evidence for her claims. I understand that people might want to question the reliability of that evidence but it's incorrect to say that there is none there in the first place.

    @Jonathan Edwards: "So the long and short of it seems to be that claims for harm following GET need to be substantiated with convincing controlled study evidence."

    Do none of research findings from Ron Davis or any of the scientists provide evidence that GET might be harmful? Is it not possible for an understanding of the biomedical nature of disease to provide reliable evidence for whether a treatment may be beneficial or harmful? (I believe this is the type of evidence that Dr Myhill draws on heavily in making her claims).

    @mango: “I used to be very interested in Dr Myhill's advice and articles. I tried a lot of what she suggested, but sadly it didn't work for me.However, my interest in what Dr Myhill has to offer totally disappeared after listening to her talk at the Abundant Energy Summit in 2015.http://forums.phoenixrising.me/index.php?threads/the-abundant-energy-summit.39513/In her talk she made a number of statements that I found hugely off-putting, such as claiming that CFS isn't a diagnosis but a "clinical picture", that carbohydrates and vegetarianism cause CFS, and that people get CFS because they stay up all night studying and drinking buckets of coffee, eating lots of chocolate bars. She also expressed several very judgmental things about the personality traits of CFS patients as a group.She also claimed that the only ones who don't get better with her treatments are the ones who don't try hard enough or not long enough...I was shocked and felt very upset after listening to that talk.”

    I am sorry to read this @mango, I’m not surprised you felt that way. Dr Myhill absolutely deserves criticism for saying some of these things.

    @Jonathan Edwards: "Unfortunately, audit is quite often hijacked as a 'soft' way of trying to show a treatment works. 'Let's see if the treatment we have been using is making people better'."

    "If your lightbulb moment comes from lab science rather than an anecdote, as in my case, there is a place for a small preliminary trial that is not expected to provide reliable evidence that the treatment is effective but can be expected to give some idea whether the whole thing is a waste of time or whether a full trial should be set up."


    Is it not possible that the science laboratory could become hijacked in exactly the same way as you describe for an audit?
     
    Last edited: Apr 14, 2017
    PhoenixDown and Knockknock like this.
  4. Laelia

    Laelia Senior Member

    Messages:
    243
    Likes:
    444
    UK
    This was a post I wrote yesterday on the other thread:

    "I said I wanted to revisit this point later on but on reflection I can respond to it now. I have come to the conclusion that Dr Myhill is indeed guilty of promoting treatments as well founded when they are not. However, in her defence, I don't think this is intentional. I believe that in her eyes, the treatments are well founded. I would also like to point out that she is very open about what evidence there is for the treatments she recommends, as detailed on her website.

    There is much more to say about this but that's all I can manage for the time being."

    I've just gone back to edit this post adding this statement:

    [Edit: I wrote this when I was absolutely shattered. I really should avoid posting comments when I am in this condition. Now that I am more awake I am doubting the conclusion that I came to that Dr Myhill is guilty of promoting treatments as well founded when they are not. Apologies to Dr Myhill. This is complicated and I don't think I am ready to draw any conclusions on this yet].

    I thought I should point that out because people may not see the edit in the original post.
     
    N2living likes this.
  5. TiredSam

    TiredSam The wise nematode hibernates

    Messages:
    2,283
    Likes:
    16,396
    Germany
    That was a very selective quote, you omitted my last sentence where I acknowledge that she has anecdotal evidence, but that it's not enough to be taken seriously. I suppose I could replace your bold with "if she provides insufficient evidence for her claims", but I did clarify in my last sentence which you omitted. Selective quoting and arguments bordering on semantics isn't going to get us very far.

    My full post including the omitted sentence:

     
    barbc56 likes this.
  6. Murph

    Murph :)

    Messages:
    317
    Likes:
    1,708
    In the other thread, Laelia said:
    Yes I understand that now, you all made that very clear, thank you. I wasn't trying to suggest that an audit would provide suffient evidence to prove that a particular treatment is effective. I was simply trying to suggest that it might provide some useful information. Jonathan disagrees.

    Can we all at least agree that there are some forms of useful evidence in this world that don't come from controlled trials?

    We should agree with that proposition.

    The Rituximab and cyclophosphamide pilot studies, and the 2015 Rituximab stage II study are both examples of useful evidence, and they were not controlled trials.

    They don't provide evidence of a standard that is sufficient for a national health care centre to write a treatment guideline, but that does not mean they don't have value - they can guide further research and also provide evidence for treatment outside the guidelines.

    Clinical trials are terrific, but...


    Large multi-centre randomised double blinded placebo controlled trials are terrific, but they can also be fetishised. Yes, they're the highest available standard of evidence. But they are not perfect. They can produce wrong answers via stochastic processes or operator error, (and, when they are sponsored by interested parties they can be run again and again until they produce the wrong answer the interested party likes!).

    They are also incredibly expensive. Not every thing that works has been tested in that context yet, but whether or not it works is not dependent on the test. Many things work despite a lack of trial evidence they work (it's a schrodingers cat situation but I believe we do not create the material world by measuring it!). And many things will never be tested in a clinical trial, because there is no perceived risk to them.

    The line between regular life and medicine is not as distinct as people would have us believe, and we don't rely on clinical trial evidence for a lot of health-relevant situations, like the house we live in, the specific kind of exercise we take or the specific food we eat. For these things we rely on what we may term common knowledge.


    This New York Times piece on cancer drugs is about doctors eager to use a new drug before a clinical trial is over, and their arguments that it is so obvious the drug is good they don't need that standard of evidence: http://www.nytimes.com/2010/09/19/health/research/19trial.html?

    Doctors also rely far more on received wisdom than they'd like to admit. Does that brand of thermometer measure temperature accurately? Does going round using the title doctor help patients? etc. It's only the pointiest decisions, especially pharmacological ones where there is high cost and/or high risk, where we demand a high standard of evidence that requires full clinical trials.

    For these things we rely on observation, or what we may term common knowledge.

    So while some people like to claim that anything unsupported by the highest standard of evidence lacks any evidence, they act as though that weren't true. They pursue, each day, many health-relevant interventions that are unsupported by such evidence.

    If these people tell you you're an idiot for doing something for which there is no evidence, tell em to go jump. Ron Davis ran a native American ceremony in his house for Whitney ,for crying out loud.

    Evidence is not proof


    Even when it comes to major clinical trials than have been run there is reason to continue to hold some doubt in mind.

    Some treatments might be very useful for a specific subgroup that will never be included in a clinical trial in sufficient numbers to tip the balance to statistical significance. (The most obvious example: most clinical trial participants are men).

    This means some individuals may get serious benefit from things that have been shown to be no better than placebo in populations. It's only if you believe humans are cookie-cutter identical that you think large clinical trials definitively say whether an individual will be helped or harmed by a treatment.

    I'd love for there to be a clinical trial of everything, but we don't have infinite time nor infinite money. So we will continue to operate on partial evidence.

    Relevance of all this to this thread:

    Lots of things that have not gone through large, multi centre, randomised, double-blinded, placebo-controlled trials have some evidence they work for some ME/CFS patients (LDN, ampligen, rituximab, cyclophosphamide, glutathione, Ubiquinol, florinef, antiotics, antivirals, beta blockers, pacing, dietary changes, methylation). It is not the highest standard of evidence (yet!), but that is not the only kind of evidence.
     
    Last edited: Apr 15, 2017
    ljimbo423 and Laelia like this.
  7. Laelia

    Laelia Senior Member

    Messages:
    243
    Likes:
    444
    UK
    Apologies @TiredSam if it seemed like I was being pernickety. I do think we need to be careful though over our choice of words especially when we are making accusations about people. If we are not then people can and often do get the wrong idea.
     
    Last edited: Apr 15, 2017
  8. TiredSam

    TiredSam The wise nematode hibernates

    Messages:
    2,283
    Likes:
    16,396
    Germany
    Well in that case can I be pernickety and ask who I have accused of what?
     
    barbc56 and Laelia like this.
  9. Laelia

    Laelia Senior Member

    Messages:
    243
    Likes:
    444
    UK
    Apologies again. My mistake.
     
  10. Laelia

    Laelia Senior Member

    Messages:
    243
    Likes:
    444
    UK
    Thank you @Murph for your detailed comments on this. These are some of the thoughts that I have also been having but I've not yet found the energy to put them into words.

    Thank you also for demonstrating that it is possible to link to a post on another thread. That's very handy to know! :)
     
  11. Laelia

    Laelia Senior Member

    Messages:
    243
    Likes:
    444
    UK
    Hello again @Murph. I can't seem to figure out how you neatly linked to a post on the other thread. Please can you help me with this?
     
  12. Murph

    Murph :)

    Messages:
    317
    Likes:
    1,708
    I wrote it in the other thread, then realised I was derailing that thread so I did the old ctrl+x, ctrl+v ... Not skill, just luck!
     
    Laelia likes this.
  13. Laelia

    Laelia Senior Member

    Messages:
    243
    Likes:
    444
    UK
    Thank you but I'm using a tablet so this isn't helping me! And now I'm derailing this thread! Perhaps I should post this question in the technical support forum...
     
  14. JohnCB

    JohnCB MEow

    Messages:
    234
    Likes:
    1,286
    England
    I assume you are typing using an on screen keyboard. Some will let you do these things but it may depend on the system running. I use a Windows tablet and the system lets you use several different onscreen keyboard options including ctrl and alt keys. Of course options are different if you buy a third party on-screen keyboard. I used to use an Android tablet a few years ago. I seem to recall similar options. I don't know anything much about Apple.

    If you are doing a lot of text entry on a tablet then it is worth considering getting a physical keyboard to go with it. You don't need a special proprietary keyboard. Really there are two options, a wired keyboard or a bluetooth wireless keyboard.

    All tablets have bluetooth. You need to turn this on in the tablet settings and then do the "pairing" procedure so that the tablet and keyboard are wirelessly connected. I have done this with Windows and Android. I understand that Apple is the same.

    If your tablet has a USB socket then you can use a standard USB wired keyboard. What you will need is an OTG (On The Go) cable or adaptor which connects to the mini or micro USB socket on your tablet and a full size USB to connect to the keyboard. Once again I have done this with both Android and Windows.

    I am not familiar with Apple, but I think with that you would need to use bluetooth.

    You can also connect a mouse in the same manner (except with Apple). You can have a keyboard and a mouse bluetooth paired at the same time. You can also use wired mouse and keyboard together if you connect a USB hub onto the OTG adapter. You can also get bluetooth keyboards which have built in trackpads.

    I think bluetooth would be more satisfactory generally.
     
    Laelia likes this.
  15. Laelia

    Laelia Senior Member

    Messages:
    243
    Likes:
    444
    UK
    Thanks so much @JohnCB for all your advice on this. You've reminded me that my dad gave me a wireless keyboard to use with my tablet. I've never used it because I'm always lying down when I'm on my tablet and I thought it would be too fiddly. It sounds like it could be jolly useful though so I will dig it out and have a go on it. Maybe I need to get a laptop table or something to make it easier to use while lying down. Thanks again for your advice, this is really helpful.
     
    Last edited: Apr 15, 2017
    JohnCB likes this.
  16. pamojja

    pamojja Senior Member

    Messages:
    284
    Likes:
    260
    Austria
    When one gets a major chronic disease and is prescribed a treatment, one could look up for the evidence based 'number needed to treat' (NNT) at a side like this:
    http://www.thennt.com/

    When I was diagnosed with peripheral arterial disease (PAD) I was prescribed statin and aspirin to be taken for live, and recommended a extensive vascular prosthesis for my whole abdominal aorta including arteries down the legs.

    That's the evidenced NNT for the usual treatments in CVD (of which PAD is a variety of), the disease most of us will die from anyway:

    For longer than 5 years there still isn't any data at all (despite being prescribed till your deathbed without any evidence). The kind of vascular prostheses advised to me hasn't been in use long enough, but there is the preliminary experience that 1 out of 10 have to be replaced within 5-10 years again.


    Now that is really what reliable evidence is!

    But personally to me, that evidence seems worse than Russian Roulette (where one would have at least a 5 out of 6 chance of surviving). For example with statins - prescribed to millions worldwide - where really only in 1 out of 83 it prevents premature death due to that medication (within 5 years, beyond it's everyone's guess).

    Instead of playing this worse kind of 'evidence based' Russian Roulette, I went the complete opposite and followed so-called 'quacks' like Linus Pauling's therapy. And this paid off in much better ways than evidence based medicine ever could, in that my walking disability gradually improved till a 60% government-certified disability due to it has been revoked (usually PAD is mercilessly progressing, and considered non-reversible by conventional medicine).

    Now that is the available evidence of treatments for one of the most prevalent, longterm and most widely studied deathly disease. And in my eyes it's only worth for making an informed risk/benefit analysis according to one's own proposition in taking risk. And for that I'm glad it exists.

    Then I turned around to all the natural approaches - in my case orthomolecular and herbal medicine - and choose those because I'm not that much of a risk taker :cautious:. And the evidence from reported toxic events from natural compounds is exceptionally acceptable to me (compared to any patented synthetic compound). A lack of 'evidence' here rather portraits that natural molecules can't be patented, therefore studied in double-blind studies only at billions loss, and won't happen ever.


    With acute and less multi-factorial diseases it's of course completely different again. But ME isn't. And evidence, if ever, will only be produced for patentable and more risky compounds.

    edit: typos

     
    Last edited: Apr 16, 2017
    Laelia likes this.
  17. Laelia

    Laelia Senior Member

    Messages:
    243
    Likes:
    444
    UK
    @wdb: "Dr Myhill made a very clear unambiguous statement that is not supported by any reasonable standard of evidence, the burden of proof is on her to back up the claim, if she does not have the resources to back up the claim then she should not be making it."

    @Jonathan Edwards: "The answer to that is that they should not then sell unproven treatments while writing books indicating that they are well founded."

    I gather from what you have already said @wdb that by "burden of proof" you mean a controlled trial.

    I understand @Jonathan Edwards that by "unproven treatment" you also mean a treatment that has not been subjected to a contolled trial.

    Please correct me if I have misunderstood either of you here.

    I would simply like to point out that a controlled trial is not always "proof" for the reasons explained by @Murph above.
     
    Last edited: Apr 18, 2017
  18. Laelia

    Laelia Senior Member

    Messages:
    243
    Likes:
    444
    UK
    @wdb: "A more appropriate statement would be:

    'A number of patients have reported to me significant benefits from the treatments I offer and that they have been able to get back to work and off benefits. The group of patients however met no formal inclusion criteria so may have suffered from ME,CFS,CF or any other fatiguing condition, no control data was collected so it is unknown to what extent self reporting bias may have played nor what proportion of the patients may have made similar improvements without receiving my treatments. Therefore further study is required to verify this preliminary anecdotal data.' "


    I think the problem with this statement (and I made the same mistake too when I attempted to do this) is that it omits a lot of other evidence that Dr Myhill is basing her claims on. So in that sense it really over simplifies things.
     
  19. Jonathan Edwards

    Jonathan Edwards Senior Member

    Messages:
    4,856
    Likes:
    27,027
    I think you are playing around with words a bit here, Laelia.

    By unproven in this case I mean not tested in a reliable way, and that can all sorts of things in different situations. If you have a cancer like disseminated choriocarcinoma, that has been universally fatal, then if five patients given methotrexate, without any control dummy, all survive with vanishing of the tumour, that is reliable evidence because the probability that the methotrexate had no effect can be seen to be near zero. In situations where things are less black and white properly designed trials are usually needed for reliable evidence.

    And of course a controlled trial is not necessarily proof of anything - that is what we have been writing to the Lancet about PACE about. It is a trial that calls itself controlled that proves nothing. The best way to cook a chicken may be to put it in the oven, but even if you put it in the oven you cannot be sure it is cooked - you need to stick a skewer in and make sure there is no red juice coming out. But that makes no difference to the fact that if you do not put your chicken anywhere near a cooker it will be uncooked.

    The fact that highly organised trials are often done badly and so are unreliable makes no difference to the fact that without some sort of careful assessment nobody has any idea.

    And in relation to pamojja's comment about statins - that is reliable evidence for what it reports but any sensible person can see that that is not reliable evidence for statins being worth taking. Government guidelines are rubbish and irrelevant here. Reliable evidence is ultimately what an intelligent person with experience of all the pitfalls involved can work out is reliable from common sense. It has nothing particular to do with any specific methodology.
     
    Barry53 and RogerBlack like this.
  20. Jonathan Edwards

    Jonathan Edwards Senior Member

    Messages:
    4,856
    Likes:
    27,027
    Are you sure? What other evidence does she have?
     

See more popular forum discussions.

Share This Page