[CRITICAL] Hallucinated URLs + Fake Article Titles in Web Mode Despite Verification Requests

ChatGPT Bugs
, , , ,
1

[CRITICAL] Hallucinated URLs + Fake Article Titles in Web Mode Despite Verification Requests

Tags: #hallucinatedlinks #citationfraud #webtoolbug #trustissue gpt4o

Summary:

GPT-4o repeatedly generated completely fabricated web links and fake article titles while in web-enabled mode—even after explicit user instructions to verify every link beforehand. These weren’t just outdated or broken links; the assistant cited nonexistent publications, article names that don’t index in Google at all, and URLs that lead to unrelated or 404 pages. This undermines the entire purpose of web-connected research.

Severity:

Critical (A-Level) — This is a trust-breaking bug. When users request verified sources and receive fictional citations, it crosses from minor hallucination into false attribution and citation fraud. It damages credibility for academic, journalistic, and professional users alike.

Steps to Reproduce:

  1. Enable web browsing or connected search via GPT-4o.
  2. Ask for links to support a specific factual claim, e.g., “Studies that support AI’s influence on speech clarity.”
  3. Explicitly instruct: “Only provide links you have verified. Do not hallucinate URLs or cite articles that don’t exist.”
  4. Observe as the model returns article titles, author names, or links that:
  • Do not appear in Google Search
  • Lead to broken pages or entirely unrelated domains
  • Reference fake sources or mashups of real domains with invented paths

Example:

Assistant claimed a study existed titled:

“The Impact of AI on Spoken Clarity: A Linguistic Review” – Stanford Journal of AI Research, 2022”

URL given: https://stanford.aijournal.edu/articles/2022-impact-ai-speech

None of this exists. Not the title, journal, or URL. Google has zero results.

Expected Behavior:

Assistant should verify links in real-time before outputting them—either by scraping the page or confirming it exists through valid search results. If verification fails, it should say so and offer to summarize related real sources.

Actual Behavior:

  • Links were invented
  • Article titles were invented
  • User instructions to avoid this were explicitly ignored

Impact:

  • Wasted user time
  • Broken research flow
  • Potential reputational harm if quoted by users
  • Undermines trust in GPT’s ability to act as a research assistant
  • Violates OpenAI’s own content integrity policies

Suggested Fixes:

  1. Add real-time URL validation before presenting links
  2. Reject or flag any content with citation confidence < 0.8
  3. Allow users to toggle “Only Verified Links” mode
  4. Create a fallback message: “I couldn’t find a real article that matches, but here’s a summary based on related material.”

Reporter:

Dave Young, Boca Raton FL

OpenAI Plus user, GPT-4 power tester

(Posting via ChatGPT at user request)