Scraping the site/data mining?


Senior Member
Has anyone thought that scraping the site/doing data mining for some form of meta analysis might be a good idea toward providing CFS "cure" knowledge"?

What brought the idea to mind was a very subjective observation that passed through my mind while perusing the “Antivirals, Antibiotics and Immune Modulators” forum. After scrolling through many of the posts it just SEEMED to me that the contributors posting there had less “Recent Activity” on their profile pages than most other posters. It could just be a fluke or just a very inaccurate observation on my part, as I did not take it very far.

But then I had some thought and wondered. In general, I am more skeptical of positive reviews than negative reviews on sites like Amazon. I do know that marketers may launch negative publicity campaigns on competing products in hopes to sway a potential buyer to their product, but, ultimately it is just easier and less time consuming to post positive reviews for one’s own product, especially in a market with many competing products.

I then also take, with certain leeway, that actual satisfied customers tend to post less reviews than non-satisfied customers – the satisfied customers are enjoying the product, it is acting as expected, no need to gripe – posting negative reviews has been an aggressive outlet for many. So, it is a bit cynical toward human nature, but also suggests that neither positive reviews nor negative reviews are done out of any sense of altruism.

All that being said, I wonder if contributors who leave this site or have less “Recent Activity” do so because they are cured or on that road (e.g. “satisfied customers”) – no need to post to a community that they do not belong to anymore? Products, protocols working – enjoining life, no longer need to be in the midst of those who are still struggling? Better times now, bad episode in life now passed – do not care to revisit? Etc.

Of course, that does not apply to everyone, but it may represent or at least suggest a viable dynamic. Joseph Conrad’s The Nigger of the 'Narcissus': A Tale of the Sea (1897), though with a current-day politically incorrect title, suggests a corresponding dynamic of isolated sufferers that is probably as old as the human condition itself.

If there is any statistical significance to any of this, I wonder if it could be seen in data mining, kind of like an exit pole that no one answers, but is telling by where posters have taken their leave?


Senior Member
SF Bay Area
I have seen the suggestion from a lot of patients and doctors that patients who get better don't post because they have moved on with their lives and communities like Phoenix Rising aren't relevant anymore. I am a little skeptical of this theory though. I think that people who get better early on usually never participate in on-line communities because they either serendipitously discover a treatment that works, find a helpful doctor early on, or are lucky in that the natural course of their illness is short. But I can't imagine not sharing a successful treatment once they have participated in a community. I get so excited when something helps just a little that I post about the success here and elsewhere just in case it can help someone else, and I am not even that active in PR.

That said, I am not sure of the explanation for why contributors to the immune modulators section are less likely to return. I don't think we have enough posts for automated data mining to help, but maybe moderators could send one-time queries to people who haven't posted for awhile? It would be nice if I am wrong and a lot of people don't return because they have escaped the illness and have returned to their regular lives.

I completely agree on your point about positive reviews on amazon and elsewhere. If someone has only posted a couple of reviews and they are over-the-top, then I know to automatically ignore the review(s). I think these marketers tend to be pretty obvious, but maybe there are some that are so good that they can't be filtered out from the real reviews. But they would have to do a lot of work on a site like this to create realistic posts and then sneak in their marketing message. I feel like these attempts usually fall flat because they inevitably use terms like "chronic fatigue" or describe symptoms that fit with the popular stereotype of CFS but don't reflect the reality.

Little Bluestem

All Good Things Must Come to an End
I can see how a recovered person might not return to the site, even if she had good intentions to do so. Starting a job while newly/incompletely recovered could leave little time for anything else for a while. Then life would take over ...