Perhaps but we are not in a legal setting here. The people who decided that requests were vexatious were not judges in a court.
Actually it almost certainly is intended in the legal sense. If it was a matter of "You can withhold data if you're annoyed by the request", it's completely subjective and everyone could use it. The legal definition has the additional requirement of bad faith, which turns it into a meaningful concept which can be applied to data requests and refusals.
Additionally, the respondents (QMUL) aren't making a judgement. They're making an argument: we won't send you data because we believe the rules don't require it. The same thing happens in other settings ... it's not in court, but people are basing their decisions on what they understand of the law. Then, when the other party disagrees, they go to court. But it is a legal definition which people are relying on to inform their decisions.
And the forum for these "vexatious" requests and denials is very legalistic. There are laws or regulations saying when data must be released, and exceptions for that. It might be a bit odd to think of rules and regulations as being a legal subject, but we learned about plenty of them in law school. In the case of the FOI requests, lawyers are often involved, are arguing legal points based on the rules and precedent, and it's going to a tribunal (judges) on appeal.
I dare say FOIAs are entirely a legal matter.