Citation Patterns in AI-Generated Text

Citation Patterns in AI-Generated Text

Reference and citation indicators that suggest AI involvement. These patterns range from clear hallucination to subtle formatting anomalies.

Broken External Links

The Pattern

If a new article has multiple citations with external links, and several return 404 errors or don't exist, this is a strong AI signal.

Key Factors

Links are more likely AI-hallucinated if:

• They return 404 errors
• They're not found in Internet Archive or Archive Today
• The domain exists but the specific page doesn't
• Multiple citations from the same source are broken

Why This Happens

AI generates plausible-sounding URLs based on patterns in training data, but the specific pages don't exist.

Example Pattern

A draft cites three New York Times articles, all with realistic-looking URLs:

• nytimes.com/2023/05/15/technology/ai-research.html (404)
• nytimes.com/2023/08/22/business/startup-funding.html (404)
• nytimes.com/2024/01/10/science/quantum-computing.html (404)

The domain is real, the URL structure is correct, but the articles don't exist.

Detection

1. Check all external links in new articles
2. Note how many return errors
3. Search Internet Archive for any trace
4. Multiple non-existent URLs = strong AI signal

Invalid DOIs and ISBNs

DOI Verification

DOIs (Digital Object Identifiers) should resolve to actual papers. Unresolvable DOIs suggest hallucination.

ISBN Checksums

ISBNs have built-in checksums. Invalid checksums mean the ISBN is incorrect or fabricated.

ISBN-10 checksum: Sum of (digit × position) should be divisible by 11 ISBN-13 checksum: Alternating 1× and 3× multipliers, sum divisible by 10

Citation templates display warnings for invalid ISBNs.

Example of Hallucinated Academic Citations

ChatGPT generated this passage about Ohm's Law with citations:

"Ohm's Law applies to many materials and components that are 'ohmic'... However, it does not hold for non-linear devices like diodes or transistors [2][3].

References: 2. M. E. Van Valkenburg, 'The validity and limitations of Ohm's law in non-linear circuits,' Proceedings of the IEEE, vol. 62, no. 6, pp. 769–770, Jun. 1974. doi:10.1109/PROC.1974.9547 3. C. L. Fortescue, 'Ohm's Law in alternating current circuits,' Proceedings of the IEEE, vol. 55, no. 11, pp. 1934–1936, Nov. 1967. doi:10.1109/PROC.1967.6033"

Problems:

• Both DOIs lead to completely different papers
• C. L. Fortescue died 30+ years before the supposed 1967 publication
• Vol 55, Issue 11 doesn't contain any matching article

Detection

1. Click DOI links - do they resolve to the claimed paper?
2. Check ISBN checksums
3. Verify author names against publication dates
4. Cross-reference journal issue contents

Hallucinated References

Complete Fabrication Signs

• Author names that don't exist in academic records
• Book titles that return no search results
• Journals with plausible names that don't exist
• Page numbers that are obviously wrong (e.g., "pp. 1-500")

Plausible But Wrong

AI often generates references that:

• Use real journal names with fake articles
• Cite real authors with papers they didn't write
• Have correct format but fictional content

Book Citations Without Page Numbers

AI cites books without page numbers, making verification impossible:

"Dorf, R. C., & Svoboda, J. A. (2010). Introduction to Electric Circuits (8th ed.). ISBN 9780470521571."

A real book exists, but which pages support the claim? This pattern suggests AI generated a "plausible" citation without actually reading the source.

Named References Declared But Unused

The Pattern

References defined in a <references> section but never called in the article body produce cite errors:

<references>
<ref name="source1">https://example.com/article</ref>
<ref name="source2">https://example.com/other</ref>
</references>

Result:

Cite error: A list-defined reference named "source1" is not used in the content

Why This Happens

AI may generate a list of sources separately from body text, without properly linking them. Or the user asked AI to "add citations" and it added the list without integrating them.

Detection

Look for the cite error warnings that appear when references are declared but unused.

Incorrect Reference Formatting

Markdown-Style Links in Citations

According to a [G1 article](https://g1.globo.com/example) and as noted by [RG/UOL](https://siterg.uol.com.br/example)...

Source-by-Source Attribution in Body

AI over-emphasizes sources in body text:

"As highlighted by EBC Rádios, she has been recognized as one of Brazil's emerging independent artists."

Human Wikipedia editors typically provide inline citations or no attribution for uncontroversial facts.

Wrong Citation Template Syntax

AI may produce malformed citation templates:

<ref>{{cite web
 |title=Article Title
 |url=URL  <!-- Literal "URL" as placeholder -->
 |website=Website Name
 |date=2025-XX-XX  <!-- Placeholder date -->
 |access-date=2025-XX-XX
}}</ref>

Escaped Quote Marks

AI may incorrectly escape quote marks in reference names:

<ref name=\"source\">...</ref>

Instead of:

<ref name="source">...</ref>

URL Placeholder Patterns

Literal "URL" Placeholder

AI leaves "URL" as literal text:

|url=URL
|website=Example Website

Placeholder Dates

Incomplete date fields:

|date=2025
|access-date=2025-XX-XX

Detection Searches

insource:"2022-xx-xx" OR "2024-xx-xx" OR "2025-xx-xx"

Footnote Marker Patterns

↩ Character

Some AI interfaces use the ↩ character for footnotes:

KLAS Research. (2024). Top Performing RCM Vendors 2024. https://klasresearch.com ↩ ↩2

Numbered Brackets Without Links

AI may output [1], [2] markers that don't actually link to references:

"The results were significant [1] and confirmed previous findings [2]."

But [1] and [2] are plain text, not linked to any references.

Non-Existent Templates and Categories

Hallucinated Infoboxes

AI creates plausible-sounding but non-existent infobox templates:

{{Infobox ancient population
| name = Example Culture
...
}}

When the actual template is:

{{Infobox archaeological culture
...
}}

Category Name Errors

AI hallucinates slightly wrong category names:

AI outputs: [[Category:American hip hop musicians]] Actual: [[Category:American hip-hop musicians]]

The hyphen difference makes it a red link.

Detection

• Check for red-linked templates in drafts
• Check for red-linked categories
• Early revisions may show deleted broken categories

Over-Attribution Pattern

The Pattern

AI painstakingly attributes every fact to named sources in body text, even for trivial information:

"The restaurant has also been mentioned in ABC News coverage relating to incidents in the surrounding precinct, underscoring its role as a well-known late-night venue."

Human editors would typically just cite this with a footnote, not emphasize the source in prose.

Why This Happens

AI models trained on Wikipedia content see citation-heavy text and over-generalize the pattern, attributing even minor facts explicitly.

Citation Verification Checklist

When checking citations:

• Do external links resolve to actual pages?
• Are broken URLs archived anywhere?
• Do DOIs resolve to the claimed papers?
• Do ISBN checksums validate?
• Are there declared references that produce cite errors?
• Are there placeholder values (URL, XX-XX dates)?
• Are templates and categories real or red-linked?
• Is there over-attribution of minor facts?
• Are book citations missing page numbers?
• Are reference names properly formatted (no escaped quotes)?

Confidence Levels