Caesin and gluten I avoid pretty much entirely (though after being very very strict for the last couple of years, I'll admit I recently started to eat a little once in a blue moon - gluten in soy sauce or a tiny bit of goat cheese on a special occasion, for example). Other food sensitivities I cut out entirely for about a year, but now eat every once in a while. Cutting out everything I was sensitive to in any way meant eating a very limited diet for that time, and I figured in the long run it was probably the lesser of two evils to put them back in sometimes.
I think there are pros and cons on both sides, and I've heard both from experts (avoid completely for at least six months, or rotate and just don't have them more than every 4 days). On the one hand, if you're still triggering that immune response sometimes, you may not give it a chance to calm down and get better. On the other hand, if you go without a food entirely, instead of calming down your body could freak out more dramatically when you try to eat it again.
I've experienced both. Giving my body some time off of eggs, for example, means that I can eat an egg without too much trouble once in a while. But going off of milk meant that my body lost the desensitization it apparently had and now I have a full blown allergic reaction to it with hives, etc. Maybe there is no great solution.