I think your first guess was right, actually. I got ME in my first year at university. My marks dropped to half of what they had been at secondary school. Instead of looking at my academic history and realising that this was an incredibly intelligent student with medical problems, they just assumed I was a bit thick.
Similarly, when I went to see a memory specialist, many years ago when my memory wasn't nearly as bad as it is now, she made no attempt to find out what my IQ was prior to my becoming ill. She selectively interpreted the results to decide again that I'm just a bit thick. This involved ignoring tests where I couldn't even begin to answer the test, e.g. giving me a paragraph to read, taking it away, asking me to write it down, and finding that I had no idea what the subject had even been, let alone the details. Those memory specialists only deal with elderly people with Alzheimers and evidently have no clue how to deal with someone younger with other memory problems. I had most of my life missing and they failed to notice a problem.
I now have a friend who had a traumatic brain injury when she was 14, which caused total and permanent amnesia as well as continuing problems with various cognitive functions. You'd think that something that massive would be obvious - she didn't know her own name or recognise her family, and the memories of her former life never returned. She didn't even get diagnosed for nearly a year. Doctors are really not well-equipped in this area.