arXiv’s One-Year Penalty for Fake AI Citations: Run This Audit Before You Submit

Short answer: If AI touched a literature review, citation list, technical report, or research post, run a citation audit before publishing. Every reference needs a real source record, and every cited claim needs to match what the source actually says.

Answer first: If you use an LLM anywhere near a literature review, related work section, technical report, whitepaper, or research blog post, do not submit until every reference has been checked against a real source record and every cited claim has been checked against the paper it cites. That is the practical lesson from arXiv’s reported one-year penalty for unchecked AI-generated research errors.

This is not an AI ban. The core issue is unchecked output. Nature reported on May 19, 2026 that arXiv is banning researchers from posting manuscripts for one year if a submission contains AI-hallucinated references or other incontrovertible signs that generative AI output was not carefully checked. Nature also noted debate over whether this is the right enforcement approach, including concerns around fairness and responsibility. But the operational takeaway is much simpler: authors need a verifiable citation-audit workflow before anything is submitted.

The timing matters because a new arXiv paper gives the problem a scale estimate. In LLM hallucinations in the wild: Large-scale evidence from non-existent citations, submitted on May 8, 2026, Zhenyue Zhao, Yihe Wang, Toby Stuart, Mathijs De Vaan, Paul Ginsparg, and Yian Yin audited 111 million references across 2.5 million papers in arXiv, bioRxiv, SSRN, and PubMed Central. The authors report a conservative estimate of 146,932 hallucinated citations in 2025 alone, with sharp rises after widespread LLM adoption. They also report that hallucinated references disproportionately assign credit to already prominent and male scholars, and that moderation and publication processes catch only a fraction of the errors.

That means this is not just a formatting problem. A fake citation can damage a manuscript, distort credit, mislead downstream readers, and put every listed author in a hard conversation with a repository, journal, conference, employer, or funder.

What changed at arXiv?

The reported rule targets obvious evidence that authors did not check AI-generated material before submission. TechCrunch reported on May 16, 2026 that authors can face a one-year arXiv ban, followed by a requirement that later arXiv submissions first be accepted by a reputable peer-reviewed venue. TechCrunch also emphasized the important distinction: this is not an outright prohibition on LLM use. It is an insistence that authors take responsibility for content, regardless of how it was generated.

Ars Technica reported in May 2026 that the enforcement discussion came from Thomas Dietterich and that examples include hallucinated references and leftover LLM meta-comments. Those examples are useful because they describe the danger zone: not an ambiguous disagreement about interpretation, but clear evidence that nobody checked the machine output.

arXiv is not peer review. Its content moderation guidance says submissions are moderated, but that the moderation process is not a peer-review process. In other words, arXiv moderation is not your safety net. The author team is.

The real risk: citation-shaped text that looks plausible

LLMs are good at producing reference-shaped objects: real author names, plausible paper titles, familiar venues, and reasonable years. That is exactly why hallucinated citations are dangerous. They often look boring enough to survive a quick skim.

The failure usually appears in one of five forms:

The pre-submission citation audit: a workflow that actually works

Use this workflow for arXiv submissions, journal manuscripts, conference papers, technical reports, public whitepapers, and research-heavy blog posts. It is intentionally stricter than a normal bibliography pass because the risk is no longer just embarrassment. It is reputational and procedural.

Step 1: Freeze the bibliography before the final audit

Set a reference freeze date. After that point, no new citation can be added without a mini-audit. This prevents the common last-minute failure where an author asks an LLM to add related work during final edits and nobody checks the new references.

Step 2: Export a clean reference ledger

Create a spreadsheet with one row per reference. Include these columns:

Step 3: Resolve every reference through at least one authority

For each citation, search by exact title in at least one authoritative source. For biomedical work, use PubMed and publisher records. For computer science, use arXiv, DOI records, ACL Anthology, OpenReview, DBLP, publisher pages, and conference proceedings. For social science, use SSRN, DOI records, publisher pages, and institutional repositories. Google Scholar and Semantic Scholar are useful discovery layers, but do not treat a search snippet as final proof.

The pass condition is simple: the title, authors, year, venue, and identifier must point to the same real work. If any of those fields cannot be reconciled, mark the row as suspicious.

Step 4: Check the cited claim, not just the citation

A reference can exist and still be misused. For every citation attached to a substantive claim, open the source and verify that it supports the sentence. Use a four-point claim check:

This is the same principle behind Tovren’s analysis of source quality and claim fidelity in Google AI Overviews: a citation is only useful if it supports the actual claim being made.

Step 5: Search for LLM residue across the full manuscript

Run a literal text search for phrases that should never appear in a submitted document. Include terms such as: as an AI language model, here is a summary, placeholder, insert citation, verify this, illustrative data, fill in the real numbers, I cannot access, generated by, and would you like me to. Also search comments, track changes, captions, tables, figure notes, appendices, supplementary files, and BibTeX notes.

Do not limit this step to the main PDF. Leftover LLM comments can hide in LaTeX source, Markdown drafts, Word comments, Zotero notes, spreadsheet tabs, or figure alt text.

Step 6: Assign coauthor sign-off

Every paper needs a named citation owner and a final signer. For small teams, one author can verify the whole bibliography while another spot-checks high-risk references. For larger labs, assign citation blocks by section. The final signer should not be the person who generated the first literature review draft with an LLM.

Step 7: Save the audit trail

Keep the ledger, exported bibliography, and final verification date with the submission files. If a question later arises, the team can show what was checked, when, and by whom. This is not bureaucracy for its own sake. It is defensive research hygiene.

Red flags that deserve extra scrutiny

Copy-paste pre-submission audit prompt

Use this prompt as a triage assistant, not as proof. The model can help you find suspicious rows, but you still need to verify references in source databases and save the audit trail.

You are a citation-audit assistant. Your job is to find risks before submission, not to make the manuscript sound better.

I will provide:
1. The full bibliography.
2. A list of in-text citation sentences or paragraphs.
3. Any available DOI, arXiv ID, PMID, ACL Anthology ID, OpenReview link, publisher URL, or official source URL.
4. Optional notes from Zotero, BibTeX, EndNote, or my reference ledger.

Rules:
- Do not invent citations.
- Do not add replacement citations unless I provide a verifiable source record.
- If you cannot verify a reference from the information provided, mark it as NOT VERIFIED.
- If a source exists but does not support the cited claim, mark it as CLAIM MISMATCH.
- If title, authors, year, venue, DOI, arXiv ID, or URL conflict, mark the exact conflict.
- Treat famous author names, generic titles, dead DOIs, missing venues, and unsupported broad claims as high risk.
- Search for LLM residue such as placeholders, chatbot meta-comments, illustrative data notes, and instructions to insert real results.

Output:
Return a table with these columns:
Reference ID | Citation text | Verification status | Metadata conflicts | Claim support status | Risk level | Required fix | Human reviewer

After the table, list:
1. References that should be deleted unless verified.
2. Claims that must be rewritten.
3. Citations that need a second human reviewer.
4. Any text that looks like unchecked LLM output.

Use these status labels only:
VERIFIED
METADATA MISMATCH
CLAIM MISMATCH
NOT VERIFIED
POSSIBLE LLM RESIDUE
NEEDS HUMAN REVIEW

The final pre-submission checklist

• Every bibliography item resolves to a real source record.
• Every DOI, arXiv ID, PMID, URL, conference link, or publisher page has been opened and checked.
• Title, authors, year, venue, and identifier match the official record.
• Every substantive cited claim has been checked against the cited source, not just the abstract snippet.
• No citation was kept only because an LLM suggested it.
• No LLM meta-comment, placeholder, or instruction remains in the PDF, source files, figures, tables, supplement, or metadata.
• Late-stage references added after the bibliography freeze have been separately audited.
• At least one coauthor who did not generate the first AI-assisted draft has reviewed the final reference ledger.
• The audit ledger, final bibliography, and access date have been saved with the submission package.

What to do if you find a hallucinated citation

Do not quietly swap in a new source that you have not read. Fix the argument first.

For labs and research managers: make citation audits part of submission governance

Individual vigilance helps, but labs need process. Add a citation audit to your lab’s submission checklist, especially for graduate students, rotating collaborators, and multi-author papers where nobody owns the full bibliography.

A practical lab policy can be short:

• Any AI-assisted literature review must disclose which tool was used internally to the author team.
• AI-generated references are not allowed into the bibliography until verified in a source database.
• Every submission has a citation owner and a final sign-off owner.
• All final references are frozen before the last writing pass.
• Claims based on citations must be checked for support, not merely for source existence.
• The final audit ledger is stored with the submitted version.

This pattern is similar to agent governance in software work. The problem is not that a model helped. The problem is unbounded output entering a high-trust artifact without controls. Tovren has covered this same control failure in overeager coding agents editing out-of-scope files, runtime governance for AI agent pilots, agent benchmark design, MCP server tool-access audits, and prompt packs for controlling agent sprawl. The shared lesson is boring but powerful: narrow the tool’s authority, log its outputs, verify before release.

For AI builders: automate the boring checks, but fail closed

If you are building a writing assistant, literature-review agent, or research workflow, add reference verification as a first-class feature. The system should not merely generate BibTeX. It should resolve every reference through a reliable identifier, record the source of verification, and flag uncertain cases.

A safe citation pipeline should:

• Prefer DOI, arXiv ID, PMID, ACL Anthology ID, OpenReview, publisher pages, and conference proceedings over model memory.
• Return NOT VERIFIED when a source cannot be resolved.
• Separate source existence from claim support.
• Show metadata conflicts instead of silently normalizing them.
• Save timestamps and source URLs for auditability.
• Block export of a final bibliography when high-risk rows remain unresolved.

Do not design the tool to sound confident when it cannot verify. Design it to be annoyingly honest.

Community reaction is a signal, not evidence

There has been visible discussion in Reddit communities including r/MachineLearning, r/PhD, r/technology, r/math, and r/Physics. The useful signal is not that Reddit proves a policy. It does not. The useful signal is that researchers are debating the hard operational questions: coauthor responsibility, false positives, appeals, uneven enforcement, and whether fake references should be treated as scientific misconduct. Those are serious questions. They also do not change the pre-submission action item: verify every citation before release.

Bottom line

LLMs can help researchers search, summarize, structure, and edit. But they cannot be allowed to smuggle unverified citations into a research artifact. The new arXiv paper suggests hallucinated citations are already present at meaningful scale, and the reported arXiv penalty shows that repositories are willing to treat unchecked AI output as an author-responsibility failure.

The fix is not to panic and ban every AI-assisted draft. The fix is to add a real citation audit before submission. Freeze the bibliography. Resolve every source. Check every cited claim. Search for LLM residue. Assign human sign-off. Save the audit trail.

If your paper cannot pass that workflow, it is not ready to submit.

Source log

FAQ

Is arXiv banning all AI-assisted writing?

No. The reported enforcement focus is not AI use by itself. The issue is unchecked AI output, such as hallucinated references, misleading content, or leftover chatbot meta-comments that show the authors did not verify the manuscript before submission.

Can I use an LLM to help check citations?

Yes, but only as a triage assistant. Do not rely on model memory. Use it to flag suspicious references, metadata conflicts, unsupported claims, and LLM residue, then verify each source in arXiv, Crossref, PubMed, publisher pages, conference proceedings, or another authoritative record.

What should I do if I find a hallucinated citation before submission?

Delete it and rewrite the claim. Add a replacement citation only after you have found and read a real source that supports the revised sentence. Save the correction in your audit ledger.

Who is responsible if a coauthor or AI assistant added the fake reference?

The author team should assume responsibility for the final submitted text. That is why every submission needs a named citation owner, a final reference freeze, coauthor sign-off, and a saved audit trail.