I'm not so sure we can make that assumption. We know that different people have different risk factors for certain diseases due to unique gene profiles and whether or not certain genes we have are expressed or not. It may very well be that "X" might cause several disorders. One person may get cancer, someone else an autoimmune disease and a third a heart problem from the same agent.
For example, the Epstein Barr virus is best known for causing mono but it can also cause a cancer called Burkit's Lymphoma.
It would be very interesting if the following study was conducted:
1. Households having several members with different disorders are enrolled in the study.
2. Each member living there full time (including pets) has blood, urine, feces etc taken. Everything from every known bug, to genetic profiles, to chemicals in the blood etc tested for.
3. The home tested for things like mould, leaky appliances, chemical levels in the air...
4. Food diaries analysed.
5. Personal care products (soap, shampoo etc) and cleaning products compared to chemical levels in air and blood ...
Analyse all the data and who knows what connections might be revealed! Of course it would take a HUGE amount of money but I'd sure be interested in the results. I'm sure there would be some very big surprises.