Gpt-4o-search-preview searching linked pdf:s - any ideas?

API
1

Hi!
I’ve built an assistant for helping parents to kids with special needs with their communication with the schools and authorities, and it is working quite well - with the help of the Assistant API. I’ve also ensured it knows about swedish educational laws, the right context and all. Quite neat.

Now I’m in the “next phase” supercharging it with additional context by giving it function calls to read the county’s protocols from the decisions bodys, the inspector general’s visits etc. That’s where gpt-4o-search-preview comes into the equation (using a separate chat completions call besides the main thread assistant).

I have function calling working to allow me to ask “Please find the protocols for the Solna county for the last year”, and it knows which website to visit, and correctly identifies the meeting dates and all.

BUT, ALL the links to the actual protocols (in pdf) are hallucinated! I really can’t tweak it to return correct url:s to the actual pdfs - they are all 404’s. (I’ve tried to increase the context from medium to high as well, no change).

I ultimately want these protocols to be downloaded and then fed into the main assistant thread through the filesearch method, so the main assistant thread can use the knowledge in the continued dialogue with the parent.

I’m kind of stuck - any good ideas on how to (re)structure this approach? Help would be greatly appreciated! I’m having a lot of fun with building these tools - fantastic capabilities indeed!

The system prompt I’m using with gpt-4o-search-preview is: “You are a search assistant using search-preview to find specific information and valid links from specified websites. Before providing a URL in response, ensure it is a valid link that will return a PDF file for further analysis. Return the result according to the specified JSON schema. If any information is missing (e.g., mentioned, date, title, or URL), indicate ‘Not found’ for that field. The links to the protocols will later be used to download the protocols as PDFs for further analysis, so they must be valid and exist on the website. If a URL cannot be verified as valid, indicate ‘Not found’ for that field. You must NOT generate or guess URLs that do not exist on the website. If you cannot find a valid URL, indicate ‘Not found’. If you cannot find any protocols, return an empty list.”

2

I’m getting inconsistent results from the 4o-mini-search-preview, with some URLs missing from a response (but page titles and topics mentioned) and some URLs hallucinated.

Note I’ve been primarily trying it via /responses instead of chat completions, and using mini with medium context.

I may try shortening my prompt in hopes that less is more, so to speak.

If I can’t solve it front end I’ll need to augment it with some other back end link checker (likely something slower or less affordable than OpenAI’s) but even checking links is tricky; often it hallucinates legit sites that gracefully provide several http 200 response codes, even while a 404 is buried with the body of the page. Plus they have legit meta titles and descriptions, so validating links requires crawling the pages for valuable content … which sort of defeats the purpose of the OpenAI search API.

1 Like
3

Thanks for confirming and responding to the issues. I see the same, that the hallucinated URLs really are ”of some quality” - but still hallucinated…

I use Grok to help me write code, and it suggested - as you do - to build a link verificator, which in my case is easier since these links should point to a pdf file. But 4o-mini-search-preview got lazy, found 1 pdf, and then reused it over and over again…. :joy:

Anyway, hope this issue with links can be resolved in the future 4o-mini-search (w/o -preview). Also, it would be fantastic if it even could traverse linked pdfs automatically, and process them with ‘filesearch’!

Still, amazing capabilities here!
/Christian

4

Yes it’s a good start but not reliable for me yet.

I’ve also gotten the repeated links over and over.

Currently researching how to leverage the Brave search instead, because it offers a generous free tier. The priority for me is high quality search results, and after testing for hundreds of thousands of tokens I haven’t gotten reliable results from the OpenAI 4o mini search preview yet. I’m sure they’ll figure it out eventually.

I’ll probably go up to 4o and large context just to try it out, but at that cost it competes less with more established search APIs, if my math is correct.

Then again it all may be working flawlessly in a couple weeks. :joy:

5

Thanks for the pointer to Brave, will take a look at it as an alternative!

Good luck with your project!
/Christian

6

@christian.wettergren

How about putting a prompt as a part in or into any of your next steps or components of your app, like this:

You are a helpful assistant that checks the validity of PDF URLs before presenting them. When a user provides a list of links or asks for documents, verify each link using a HEAD request before including it in your answer.

Only present URLs that return a 200 OK status. If a link fails validation, exclude it and notify the user.

THEN, you can either tell it to use the python environment - or if you’re not using ANY of the ONLINE OpenAI ChatGPT environment, which I assume you don’t, deriving from your description you can add a python component for checking a link:

import requests

def verify_url(url):
    try:
        response = requests.head(url, allow_redirects=True, timeout=5)
        return response.status_code == 200
    except:
        return False

urls = [
    "https://example.com/file.pdf",
    "https://county.gov/documents/2024_protocol.pdf"
]

valid_urls = [url for url in urls if verify_url(url)]
valid_urls

Something like this.
OR, you use something programmed to find the links. Whatever helps the most.

It’s always a little bit of research and tweaking parts and so on required.
If you do more in the prompt and ask it, for example by creating a meta prompt first. Or go for an approach like this is left to your gusto.

Kudos to trying and not giving up. :slight_smile: You’ll get there, Christian. :slight_smile: