Outdated link citations with responses api and search tool

[willem.dhaeseleer]

Using gpt-4.1 with the responses api consistently returns outdated link citations. Is there an easy way to ensure the api does not return outdated links ?

[edwinarbus]

Do you happen to have any examples you could share?

[_j]

@willem.dhaeseleer

Is there an easy way to ensure the api does not return outdated links ?

AI models will frequently fabricate internet addresses and citations. They predict what an answer might look like.

You don’t say if it is precisely “web search” you discuss, or if the links are outdated by search being 12 hours behind the latest news… or outdated by never having existed. The former seems to be a behavior of the web search, a cache with a long TTL.

To guard against purely bad links that are 404, you could do a HEAD request yourself or a bytes range 0-1024 and get the status of any links the AI returns, either giving feedback in the UI about the bad created link, or auto-prompt for a correction if the AI is capable.

[willem.dhaeseleer]

Here is an example system prompt.

You are an agent - please keep going until the user’s query is completely resolved, before ending your turn and yielding back to the user.
Only terminate your turn when you are sure that the problem is solved.

When the user asks a question about EarnIn or EarnIn products search https://help.earnin.com/ for web pages that have the answer and reply with a link to the relevant page.

Then, you can ask any question that can be answered on our public help center:

https://help.earnin.com/hc/en-us/articles/213412087-How-does-the-app-work

Example:

How does the app work ?

The bot returns this link:

https://help.earnin.com/hc/en-us/articles/1500002244102-How-does-the-EarnIn-app-work-

Instead the working link is this:

https://help.earnin.com/hc/en-us/articles/213412087-How-does-the-app-work

This seems to be happening for almost all of the articles, rarely the link that is returned is the actual working link.

Is there a way to at least know when this link was indexed ? This might help confirm if it’s a stale link or an actual hallucination.

[_j]

I’m afraid the application that you have outlined in the prompt will need your own use of functions and the function-calling feature, and by doing so, it will be able to deliver a far better product by being connected directly to your own database for knowledge.

The function itself can specify in description fields how best to employ the tool, and the function return can provide recitation-quality data.

The AI via the internal tool can only write a search query, and then gets results. It cannot directly explore web sites via a URL base on its own via the web search feature, and has instructions out of your control for how to write search result answers.

[willem.dhaeseleer]

It’s actually performing very well , the only concern is the links not being up to date

[_j]

You say “it works well” - until it doesn’t actually work out for you..

There is nothing in the API itself to really diagnose what the AI is receiving in result “chunks”, to know if OpenAI is placing bad search result links, or if the AI has done any click-thru for you to come up with hallucinations.

You can see if gpt-4o at top-p of 0 gives you any difference in what it gives the user as link for the exact same input. Change everything but the search results themselves.

You can ask the AI to issue its query as it normally would, and ask about the links. However, there are instructions against that.

OpenAI will clear the entire playground DOM when you get this quality of recitation by asking the AI nicely.

So, I would still recommend function-calling. You cannot build a robust application with tools that are capricious and out of your control, and may happily report on your competitors.

[willem.dhaeseleer]

Thank you, the rag use case here is just an example. The root of the question is, can we prevent stale links from being included when using the web search tool. We see a lot of potential in the web search capability, but incorrect links makes it hard to adopt.

[vb]

There have been previous reports of this behavior in the community but they are only a few. It is not unlikely to assume that there is something with the specific implementation causing this behavior.

In order to differentiate if there is maybe an issue with web search results for the URL vs. the final URL, maybe this snippet from the cookbook can help.
Specifically looking at the ‘print(json.dumps(response.output’ part:

Web Search and States with Responses API | OpenAI Cookbook