ME/CFS Research Chatbot

TracyD

TracyD

Messages
45
Introducing the ME/CFS Research Chatbot! It's meant to provide a reliable, accurate, and up-to-date summary of published research for the general public.

While general LLMs like ChatGPT and Claude might provide fine answers to queries about myalgic encephalomyelitis/chronic fatigue syndrome, the sources are often unreported or of unknown quality, and the chatbots are prone to hallucination and to change answers from one moment to the next.

The chatbot that I have designed is a bit different: It relies only on research abstracts from the NIH's PubMed database that address this disease, it can provide a list of the abstracts consulted, and it's designed to provide the same, 'best' answer each time a particular question is asked - including admitting ignorance, if information is lacking.

For the technically inclined: The chatbot is a RAG system using Gemini 2.0 Flash to interpret the 250 abstracts that best match a user's query. The abstracts are embedded in a vector database using Chroma. And the frontend was built with Gradio and resides on Hugging Face. The full dataset of abstracts, with metadata, is here.

If you try it, please let me know how it goes.
 
Last edited:
Norwegianlad

Norwegianlad

Messages
41
That's really amazing! I found ChatGPT is making a lot of mistakes in certain areas. Very cool you managed to make this yourself. Definitely going to use this for my ME/CFS related questions.
 
Rufous McKinney

Rufous McKinney

Senior Member
Messages
14,313
Cool. I hope somebody with some savvy will try out your system.

Maybe you need a Luddite like me, to also try it out.

I tried it out now. I got a helpful answer to the basic question of if ME people have trouble digesting fats.


What does this mean after an answer: PASSAGE 4 ????
 
Rufous McKinney

Rufous McKinney

Senior Member
Messages
14,313
I"m wondering about how this program would work if it ALSO paid attention to medical research developments in certain other key fields. If it's only ME CFS, this is very valuable. If it were to also include other neurodegenerative and neuroimmune topics, that might enhance the leads. I realize that probably becomes really open ended.

so could we ask say ChatGPT to provide an answer using ONLY NIH PubMed data base? Would that reduce some of the limitations those systems have?
 
TracyD

TracyD

Messages
45
@Norwegianlad Thanks! If my chatbot gives an out-of-date answer (for example, regarding GET), you might get a more up-to-date answer by specifying that you want it to consult sources "since 2020" (or whatever).

@Rufous McKinney: Thanks for checking it out. I'm not arguing for the utter superiority of my chatbot, but I believe that it does something useful that, at this point, the others don't. (Plus, it might look good on my resume.)

1) Like it says on the chatbot page, "It responds using information only from research abstracts that ... contain the phrase 'myalgic encephalomyelitis' or 'chronic fatigue syndrome'."

2) Someday I hope to figure out how to stop it from referring to "Passage X". This simply means that it's referring to the first, or fourth, or ... abstract out of the 250 that best match your query. The list of passages changes with each new question, so the best way to identify a particular source is to ask for a list as part of your question. For example, "What is PEM? Please provide a list of the abstracts you use, including the title and date."

3) Expanding to other fields would definitely make it run more slowly. Aside from that, it is likely that the chatbot would incorrectly apply findings from other diseases to ME/CFS. LLMs don't think and apply judgment in the way that they appear to.

4) ChatGPT, Gemini, Claude, and the rest are constantly being altered. It's possible that, if you ask one of them to limit its sources, it will do so. But it might not do it in the comprehensive fashion that my chatbot uses, and you would have a lot of trouble figuring out the truth of how it really worked. Research has shown that LLMs will do one thing and say that they did something else.

Beyond that, chatbots have differing instructions. For example, other LLMs admit 'creativity' into their answers by default - that is, they randomly vary their responses each time, even though that increases the probability of providing incorrect information. Mine doesn't.
 
Last edited:
You must log in or register to reply here.
Back