FDA Drug Approval AI Generates Fake Studies

[southwestforests]

The Fabrication Problem: How AI Models Generate Fake Citations, URLs, and References​

Nayeem Islam

31 min read
·
Jun 12, 2025

https://medium.com/@nomannayeem/the...ke-citations-urls-and-references-55c052299936

How AI Models Generate Text and References​

To understand why AI models fabricate citations and references, we must first examine the fundamental mechanics of how these systems actually work. Despite their seemingly intelligent responses, AI language models like ChatGPT, Claude, Gemini, and Grok operate on principles that are fundamentally different from human knowledge retrieval and fact verification.

Text Prediction and Pattern Recognition​

At their core, all major AI language models, whether we’re discussing OpenAI’s GPT series, Google’s Gemini, Anthropic’s Claude, or X’s Grok — function as sophisticated statistical prediction engines. These models don’t “know” information in the way humans do. Instead, they generate text by analyzing vast datasets containing billions of text samples and learning to predict what word, phrase, or sentence structure is most likely to come next based on the patterns they’ve observed.

When you ask ChatGPT for a citation about climate change research, it doesn’t search through a mental library of actual papers. Instead, it examines the statistical patterns of how citations typically appear in its training data and generates text that matches those patterns. The model has learned that citations usually follow formats like:

It then produces variations of this pattern, filling in plausible-sounding details based on what it has statistically learned about how real citations are structured.

This process is remarkably similar to how these models generate any other text; they’re applying the same pattern-matching and prediction algorithms whether they’re writing a poem, explaining quantum physics, or creating a bibliography. The model treats citations as just another form of text to be generated, not as references to actual, verifiable sources.

No Intrinsic Fact Verification​

Perhaps most critically, AI language models possess no built-in mechanism for fact-checking or verifying the accuracy of the information they generate. This is true across all major platforms — Claude cannot verify if a paper it cites actually exists any more than Gemini can confirm whether a URL it provides actually works.

These models operate on what researchers call “plausibility over accuracy” — they optimize for generating responses that sound correct and coherent rather than responses that are factually verified. When Grok generates a reference to a Harvard study on artificial intelligence ethics, it’s creating that reference because such a study seems plausible based on its training patterns, not because it has confirmed that this specific study exists in Harvard’s actual publication database.

The training process reinforces this limitation. AI models learn from massive text corpora that include millions of real citations, but they also inadvertently learn from text containing errors, speculation, and even previously AI-generated content that may itself contain fabrications. The model cannot distinguish between a real citation it encountered in a legitimate academic paper and a fake citation it encountered in a blog post or fabricated content.

This creates a fundamental disconnect: while human researchers verify citations by checking databases, libraries, and original sources, AI models generate citations through statistical inference alone. They can produce text that perfectly mimics the style and format of legitimate academic references while having no connection to actual research or publications.

Consider what happens when you ask ChatGPT for a citation about a particular research topic. The model may have encountered thousands of papers about that topic during training, but remembering the exact title, author, journal, volume, issue, and page numbers requires perfect recall of highly specific details. Unlike learning general language patterns (where slight variations don’t matter), citations demand absolute precision, getting even one detail wrong renders the entire reference useless.

This problem affects all major AI models similarly. Gemini might generate a plausible-looking DOI number like “10.1016/j.envres.2023.15847” that follows the correct format but leads nowhere because the actual DOI doesn’t exist. Claude might create a convincing journal article title with realistic author names, but when researchers attempt to locate the paper, they discover it’s entirely fictional. The models have learned the patterns of how citations should look without retaining the exact details needed to make them accurate.

URLs present an even more acute version of this problem. When Grok generates a web address like “https://www.nature.com/articles/climate-adaptation-2023-study," it’s following the statistical patterns of how Nature.com URLs typically appear, but the specific article identifier is generated through prediction rather than retrieved from memory of an actual webpage. The result looks completely legitimate, until someone clicks the link.

When you ask Claude for sources about a particular topic, the model’s primary objective is to produce a response that appears helpful and comprehensive. From the model’s perspective, providing three plausible-sounding citations that perfectly match your request is far superior to responding “I don’t have access to specific citations and cannot verify references.” The training process rewards the model for being helpful and comprehensive, not for being cautious about accuracy.

This optimization creates what researchers call “confident fabrication” — AI models generate fake references with the same linguistic confidence markers they use for accurate information. ChatGPT will present a fabricated citation using phrases like “According to a comprehensive study by…” or “Research published in the Journal of…” with no indication that the reference might be fictional.

The conversational training that makes these models so engaging also reinforces this problem. Users generally prefer AI assistants that provide specific, detailed answers rather than responses filled with caveats and uncertainty. This user preference, reflected in the training process, encourages models to generate definitive-sounding references even when they lack access to verified sources.