Some of us may not have noticeable food responses, but many of us do, and some are quite dramatic. A lot of times I'd think "This feels like I had some <whatever>", and if I checked the ingredients or detailed contents, yes, it had <whatever>. These are mostly negative responses, but I've had a few positive ones as well, such as having a fatty piece of beef and sleeping the whole night through without waking (shocking!). Other responses were more subtle, and I needed to check my journal to verify the correlation.... but never felt an actual improvement, like, distinctly noticeable, a before-and-after type delineation.
Cutting out all of a food group or chemical content might be required. If a certain molecule in milk, such as a certain fatty acid, is causing you problems, how can you know that if you insist on 3 cups of coffee a day with milk. My ME started with a type IV food sensitivity, and I had to go on a rotation diet for a while. I was surprised at how few food groups comprise the typical human's diet. It's difficult to make tolerable meals when you have to avoid entire food groups for 5 or more days.