One of the things that worry me is this sort of 'alignment' of AI to say what people want it to say, rather than what the evidence may show.
Here's a paper talking about how ChatGPT can supposedly debunk social media myths around vaccine safety, which is great, but the example they use:
The researchers asked the artificial intelligence (AI) chatbot the top 50 most frequently-asked Covid-19 vaccine questions. They included queries based on myths and fake stories such as the vaccine causing Long Covid.
Ummm, we don't know how prevalent vaccine injury may be, but talking about 'fake stories such as the vaccine causing Long Covid' is pretty concerning. We still don't understand the mechanisms of Long Covid, but there are certainly some cases of temporal association. It could say there's no evidence for, only a temporal and not causal relationship, etc - but no, they go straight to 'myths and fake stories', which is why those of us with minimized and marginalized chronic illnesses are still screwed. The establishment will beat the AI until it properly says, "those people cray-cray", but in Weasel-speak it'll spit out something about somatic symptom disorder, etc.