Best way to browse the web with the API

I want to browse the web to find out about product descriptions. I have been doing some tests with ChatGPT 4o and it works fine. I have also been reading posts about this subject (browsing the web through API) and didn’t find a relevant answer.

My questions:

  1. Does the API have new capabilities that allow users to browse for a specific subject (like a product name)?

  2. If this is false, what is the best way to browse for information? Is there any built-in function in the API or Assistants?

  3. Can anyone give me a link that shows a code doing this kind of work?

I intend to ask the AI to browse for a product, and then choose (among the 5 first results) the most complete description

If you have been working on a project like this I would appreciate if you could share your experience (even if you failed :hugs:)

Thank you very much for your help.

1 Like

You can find plenty examples of integrating the API with web search API.

I intend to ask the AI to browse for a product, and then choose (among the 5 first results) the most complete description

I would suggest looking into integrating the API with a search API using function calling
https://platform.openai.com/docs/guides/function-calling

1 Like

Hi @Lalo ! You may also be interested in checking out this cookbook that explains how to integrate Bing API with OpenAI ecosystem and create a browsing experience. Bing API in particular gives you a certain amount of requests for free each month. There are other services like Brave API as well.

3 Likes

Hi @lior.boord, thank you very much for your reply.

Yes, I have been using function calling in some other projects. I just wondered if OpenAI has evolved to some built-in solution (I guess it has not).

1 Like

Hi @platypus, thanks for the link, it looks great!

That should be the response :hugs:

1 Like

Does anyone know if it’s possible to search more than one search engine via API?

The cookbook gives you the basic structure on how to browse the web, but I found in practice I got pretty bad results from this. Web scraping the page content is difficult as most sites don’t like to be scraped. Using the requests library will result in many 403 forbidden responses, because servers can easily detect that this is not a real browser making the request. Using Selenium or similar is better, but also is quite time-consuming, and you will still get many sites detecting an automated browser.

I am really curious of how OpenAI gets around this problem with their own web browsing, and how they can be so quick. Does anyone know what they are doing differently?

2 Likes