Yes, there are days when I don't feel it's safe to drive either. I do worry about the days when I feel safe to drive, and then have the symptoms come on before I return home.
One time I visited friends for Christmas dinner. I don't know if it was a food responsible, or the mental exertion of socializing, but on the drive home, I gradually became so spaced out I didn't remember to slow down for a steep, curving, icy hill. My truck ended up about 2' off the ground on a snowbank. No harm, but I was totally surprised by how lousy my ability to drive had become. I was definitely 'driving while impaired' even though I hadn't had any alcohol.