FDA Drug Approval AI Generates Fake Studies

[Mary]

I don't usually cite The Onion as a news source, but it was what alerted me to this very real story. It appears that CNN broke the story but I can't read it on the CNN website without a subscription, but you can see a bit about it below. It's also being discussed on reddit -

https://theonion.com/fda-drug-approval-ai-generates-fake-studies/

https://gizmodo.com/fdas-new-drug-approval-ai-is-generating-fake-studies-report-2000633153

 

[Rufous McKinney]

Why would a computer generate fake? Seems like a substantial flaw in the computer's matrix.

 

[andyguitar]

Why would a computer generate fake?

It don't work properly. It's junk. But it will still be used.

 

[Rufous McKinney]

It don't work properly. It's junk. But it will still be used.

its hard to understand, unless its just a basic fact that Fact and Fiction are hard to differentiate.

Identifying A Fact......may not be possible. Which begs the question, how come we all agreed that reality is like THIS?

 

[Rufous McKinney]

Here is what AI (which I did not query) said bout AI and FACTS: emphasis added

AI identifies facts by analyzing data, recognizing patterns, and making predictions based on those patterns. It doesn't "understand" facts in the way humans do, but rather uses algorithms and statistical models to identify correlations and generate responses. Automated fact-checking tools, like ClaimBuster or Full Fact, use AI to scan text, identify claims, and compare them against known information.


This leads me to believe it is impossible. The task assumes a FACT is a measurable finite noun, you can walk around it, size it up, touch it, and go YES this is a FACT.

 

[andyguitar]

It doesn't "understand" facts in the way humans do,

Because it is not intelligent.

This leads me to believe it is impossible.

Yes it is impossible -if you mean can AI think in the way that humans (and prawns) can.

 

[andyguitar]

AI identifies facts

It cant.

 

[Rufous McKinney]

It cant.

nor can humans

 

[southwestforests]

but I can't read it on the CNN website without a subscription,

Hey Mary, this independent international agency turns out to have a page with about a dozen links to different stories about the thing, some of them might work for you,
As it happens those links do include the Gizmodo article.

https://oecd.ai/en/incidents/2025-07-23-2dc4

For those who might not know what the OECD is,

The OECD (Organisation for Economic Co-operation and Development) is a forum and knowledge hub for data, analysis and best practices in public policy. We work with over 100 countries across the world to build stronger, fairer and cleaner societies – helping to shape better policies for better lives.

The forerunner of the OECD was the Organisation for European Economic Co-operation (OEEC), which was formed to administer American and Canadian aid under the Marshall Plan for the reconstruction of Europe after World War II. The Convention transforming the OEEC into the OECD was signed at the Chateau de la Muette in Paris on 14 December 1960 and entered into force on 30 September 1961.

Since then, the OECD's vocation has been to deliver greater well-being worldwide by advising governments on policies that support resilient, inclusive and sustainable growth. Through evidence-based policy analysis and recommendations, standards and global policy networks, including close collaboration with the G7 and the G20, the OECD has helped advance reforms and multilateral solutions to global challenges. These span the public policy horizon, from the polluter pays principle, which the OECD developed in the 1970s, to PISA in education, not to mention tax transparency and artificial intelligence. Throughout its history, the OECD has striven to become more global, more inclusive and more relevant.

 

[southwestforests]

Just for fun, here's the FDA's own press release in June about the Elsa thing,

https://www.fda.gov/news-events/pre...-ai-tool-optimize-performance-american-people

FDA News Release
FDA Launches Agency-Wide AI Tool to Optimize Performance for the American People
For Immediate Release:
June 02, 2025

The U.S. Food and Drug Administration (FDA) today launched Elsa, a generative Artificial Intelligence (AI) tool designed to help employees—from scientific reviewers to investigators—work more efficiently. This innovative tool modernizes agency functions and leverages AI capabilities to better serve the American people.

“Following a very successful pilot program with FDA’s scientific reviewers, I set an aggressive timeline to scale AI agency-wide by June 30,” said FDA Commissioner Marty Makary, M.D., M.P.H. “Today’s rollout of Elsa is ahead of schedule and under budget, thanks to the collaboration of our in-house experts across the centers.”

Built within a high-security GovCloud environment, Elsa offers a secure platform for FDA employees to access internal documents while ensuring all information remains within the agency. The models do not train on data submitted by regulated industry, safeguarding the sensitive research and data handled by FDA staff.

“Today marks the dawn of the AI era at the FDA with the release of Elsa, AI is no longer a distant promise but a dynamic force enhancing and optimizing the performance and potential of every employee,” said FDA Chief AI Officer Jeremy Walsh. “As we learn how employees are using the tool, our development team will be able to add capabilities and grow with the needs of employees and the agency.”

The agency is already using Elsa to accelerate clinical protocol reviews, shorten the time needed for scientific evaluations, and identify high-priority inspection targets.

Elsa is a large language model–powered AI tool designed to assist with reading, writing, and summarizing. It can summarize adverse events to support safety profile assessments, perform faster label comparisons, and generate code to help develop databases for nonclinical applications. These are just a few examples of how Elsa will be used across the enterprise to improve operational efficiency.

The introduction of Elsa is the initial step in the FDA’s overall AI journey. As the tool matures, the agency has plans to integrate more AI in different processes, such as data processing and generative-AI functions to further support the FDA’s mission.

Prioritizing efficiency and responsibility, the FDA launched Elsa ahead of schedule using an all-center approach. Leaders and technologists across the agency collaborated, demonstrating the FDA’s ability to transform its operations through AI.

 

[Mary]

As it happens those links do include the Gizmodo article.

Yeah, that's where I found the gizmodo article. But interestingly (or not?), no other major news sources besides CNN seem to have picked this up!

 

[southwestforests]

But interestingly (or not?), no other major news sources besides CNN seem to have picked this up!

I don't keep track of national news enough to know without looking that up ...
and ...
right here right now Google found precisely 1 (one) other 3 letter network with an article,
ABC affiliate in Columbia SC,
https://www.abccolumbia.com/2025/07/24/fda-ai-tools-cites-nonexistent-studies/

 

[Viala]

My question is WHY there is 'AI' and 'drug approval' in the same sentence.

Are these guys that lazy that they need AI to do their job? Is AI even safe to let it anywhere near drug approval? I don't use AI because I would have to do a lot of fact checking and that's additional work, it's mostly useless to me, but for what they want to use it, it can be dangerous.

If AI was a drug itself it wouldn't pass any test because of the errors. Why these institutions even touch that stuff I can't understand. It's like hiring a schizophrenic that is hallucinating and they need more people to watch over it's shoulder. There is no logic in that. They want to make their lives easier so much but they use something that complicates it ten times more. Why? Because I start to think that they're doing it on purpose.

 

[Mary]

I don't keep track of national news enough to know without looking that up ...

I don't either - it's just when I google what I think to be a major story, I'll usually get several hits by the major news sources as well as lesser known ones. But there were very few hits with this story, which is concerning.

 

[bad1080]

Why would a computer generate fake? Seems like a substantial flaw in the computer's matrix.

“Anything that you don’t have time to double-check is unreliable. It hallucinates confidently,” one unnamed FDA employee told CNN.

https://en.wikipedia.org/wiki/Hallucination_(artificial_intelligence)