Yes is the quick answer to that. Apparently Olive Oil is not so much heart healthy as just less unhealthy than other fats. If Olive oil replaces a worse fat (e.g. a trans fat or something) it will improve the persons diet, but this doesn't mean adding it is always good, it is cutting out the worse fat that might be where it would do most good. Olive oil has a high enough amount of saturated fat at 14%. I think I read that a study with monkeys showed heart disease when fed a diet high in Olive Oil.
Brenda has already mentioned the problem with oils becoming rancid.
As well as this problem, and the one you mentioned Sasha about empty calories (and add to that high calorie density - that is a lot of calories for a small volume) apparently oils in the diet can reduce blood flow (which can be already impaired in ME/CFS, so I can see that logically reducing oils could help us).
This is a clip from a talk by Jeff Novick about problems with olive oil
http://www.youtube.com/watch?v=GfBKauKVi4M
and this is another one on oil http://www.youtube.com/watch?v=lbALgjmZUek
I find his stuff the most convincing. Unfortunately the best talk by him which I saw online is no longer on it, but he does have a few DVD's for sale from various places including Amazon. He has a lot of free written information here http://www.drmcdougall.com/forums/viewforum.php?f=22 but I find it easier to watch the videos.
I didn't know what to think about the no oil thing at first. I wasn't totally convinced but reduced my use initially where I knew it would be easy (e.g. "stir-fry" with water instead of oil, and just leaving it out of many recipes where it was just added in). But as I have listened to more talks, and read a bit more, I am more convinced that it could be a problem and I have gotten stricter. I have gotten some good recipe books with tips and alternatives to fats in recipes, and this has helped, and this is a good website here http://blog.fatfreevegan.com/ If you boil any veg in water you can keep the water and re-use it as stock.
It has taken me a while to learn what to avoid and what to eat (and what to buy and what to avoid). It feels more like second nature now. I think you are doing well to have cut out what you have so far, so I would just go at your own pace in making changes and learning new things, as it is hard for us to put energy into making these sort of changes.
Caldwell Esselstyn is a heart specialist that is very anti "free" oils (though he eats ground whole flax seeds for Omega 3's). As far as I can remember, from watching some of his talks, oils can damage the blood vessel walls and this damage can be cumulative over time. I have a book of his but have not read it yet (other than use some of the recipes in it).
I can definitely see that theoretically that cutting out oils could help us, as a lot of us seem to have impaired blood flow and basically some sort of vascular abnormalities, but whether it makes a big difference in symptoms for us to cut out oils I suppose we have to just try it and see. Unfortunately for us maybe we have a few things going on causing symptoms, so even if we do one thing that helps maybe it will only help a bit in terms of symptom reduction?